2018年的东京迪士尼之旅

2019年1月的清迈之旅

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

2019年10月的大阪 姬路 京都之旅

2019年6月的冲绳之旅

2018年的北海道之旅

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Flutter Widget创建时运行异步操作

样例代码

import 'package:flutter/material.dart';
import 'dart:async';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    // TODO: implement build

    return MaterialApp(
        home: Scaffold(
      body: Center(child: AsyncTest()),
    ));
  }
}

class AsyncTest extends StatefulWidget {
  @override
  _AsyncTestState createState() => _AsyncTestState();
}

int getNumber(int n) {
  if (n == 0) {
    return 0;
  } else if (n == 1) {
    return 1;
  } else {
    return getNumber(n - 1) + getNumber(n - 2);
  }
}

Future loadAsset(BuildContext context) async {
  return await DefaultAssetBundle.of(context)
      .loadString('assets/novels/0-1.txt');
}

class _AsyncTestState extends State {
  bool loading = true;

  @override
  void initState() {
    super.initState();

    loadAsset(context).then((result) {
      debugPrint(
          'consume cpu time function result is ' + getNumber(20).toString());

      Future.delayed(Duration(seconds: 1)).then((result) {
        setState(() {
          loading = false;
        });
      });
    });
  }


  
  @override
  Widget build(BuildContext context) {
    // TODO: implement build
    if (loading == true) {
      return CircularProgressIndicator();
    }

    return Text('load finished ');
  }
}


几点说明:

  1. 使用StatefulWidget 有状态Widget
  2. 在initState函数中执行异步操作
  3. 保证在已有结果的状态下调用setState

 

 

参考:

https://flutter.institute/run-async-operation-on-widget-creation/

 

阿里云 ECS 快速安装Oracle 19c

阿里云

 

 

 

操作系统使用 CentOS 7.6 64位,目前对于oracle没有任何必要使用32位操作系统了!!

磁盘在40GB系统盘的基础上增加一块ESSD云盘 大小40GB,作为存放oracle数据库的基础配置。并启用每日自动备份!

 

 

网络设置使用默认配置,用户实际使用时:

  1. 若应用服务器位于阿里云同机房,一般可以直接使用内网连接,不需要太大的公网带宽
  2. 若应用服务器不在阿里云同机房,需要使用公网连接,则需要提高公网带宽

 

无需启动3389端口

 

 

 

 

安全组使用默认配置,后续需要为ORACLE监听配置安全组网络端口。

 

最终配置报价时317元/每月!

 

最后确认订单,服务器就绪后,ssh远程登陆。

首先划分磁盘并 测试ESSD的IO, 第一块ESSD的路径一般为 /dev/vdb, 在vdb上划分1个分区,并创建XFS文件系统,挂在到  /d01目录下

 

 

fdisk /dev/vdb



Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x5d899fda.

Command (m for help): p

Disk /dev/vdb: 42.9 GB, 42949672960 bytes, 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x5d899fda

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-83886079, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-83886079, default 83886079):
Using default value 83886079
Partition 1 of type Linux and of size 40 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.



partprobe /dev/vdb

[root@iZuf6fz9mqmeexkh25fbrbZ ~]# ls -l /dev/vdb1
brw-rw---- 1 root disk 253, 17 Sep 11 12:51 /dev/vdb1


 mkfs.xfs /dev/vdb1
 mkdir /d01
 mount /dev/vdb1 /d01
 
 
[root@iZuf6fz9mqmeexkh25fbrbZ ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G   12G   27G  31% /
devtmpfs        3.8G     0  3.8G   0% /dev
tmpfs           3.8G     0  3.8G   0% /dev/shm
tmpfs           3.8G  636K  3.8G   1% /run
tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
tmpfs           768M     0  768M   0% /run/user/0
/dev/vdb1        40G   33M   40G   1% /d01


chown oracle /d01
echo "/dev/vdb1  /d01  xfs" >> /etc/fstab


 

 

测试一下ESSD的IO :

 

 

 

yum install fio


cd /d01 

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64


fio-3.1
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=11.1MiB/s,w=3744KiB/s][r=2848,w=936 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=21835: Wed Sep 11 13:48:03 2019
   read: IOPS=2847, BW=11.1MiB/s (11.7MB/s)(3070MiB/275992msec)
   bw (  KiB/s): min=10560, max=12232, per=100.00%, avg=11393.26, stdev=241.24, samples=551
   iops        : min= 2640, max= 3058, avg=2848.32, stdev=60.34, samples=551
  write: IOPS=951, BW=3807KiB/s (3898kB/s)(1026MiB/275992msec)
   bw (  KiB/s): min= 3344, max= 4256, per=100.00%, avg=3807.62, stdev=165.68, samples=551
   iops        : min=  836, max= 1064, avg=951.90, stdev=41.42, samples=551
  cpu          : usr=0.59%, sys=2.51%, ctx=951009, majf=0, minf=22
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=11.1MiB/s (11.7MB/s), 11.1MiB/s-11.1MiB/s (11.7MB/s-11.7MB/s), io=3070MiB (3219MB), run=275992-275992msec
  WRITE: bw=3807KiB/s (3898kB/s), 3807KiB/s-3807KiB/s (3898kB/s-3898kB/s), io=1026MiB (1076MB), run=275992-275992msec

Disk stats (read/write):
  vdb: ios=785887/263160, merge=0/3, ticks=13264955/4397142, in_queue=15400027, util=87.17%


可以看到 单块阿里云ESSD的随机读写  IOPS ,read: IOPS=2847 write: IOPS=951        还是不错的。

 

开始安装oracle 19c  , 首先上传介质到服务器/root目录

使用浏览器下载以下2个介质并上传到服务器目录

https://download.oracle.com/otn/linux/oracle19c/190000/oracle-database-ee-19c-1.0-1.x86_64.rpm
https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/oracle-database-preinstall-19c-1.0-1.el7.x86_64.rpm


su - root 
yum -y install git 
git clone https://github.com/macleanliu/ora-easy-deploy
yum -y localinstall oracle-database-preinstall-19c-1.0-1.el7.x86_64.rpm
rpm -ivh oracle-database-ee-19c-1.0-1.x86_64.rpm
bash ora-easy-deploy/create_db.sh


 

bash ora-easy-deploy/create_db.sh ,执行该脚本输入 数据库名 ,数据库目录和SYS/SYSTEM密码等,后自动建库。

 

安装完成后,执行:

 

 

#设置ORACLE OS账户密码
su - root
passwd oracle                     
su - oracle

lsnrctl start 
sql
alter system register;
alter system set "_optimizer_aggr_groupby_elim"=false;
alter system set "_drop_stat_segment"=1;
alter system set "_common_data_view_enabled"=false;
alter system set optimizer_adaptive_features=false;
alter system set "_optimizer_dsdir_usage_control"=0; 
alter system set "_enable_automatic_sqltune"=false scope=both; 
alter system set "_serial_direct_read"=false;
alter system set "_nlj_batching_enabled" = 0; 
alter system set "_optimizer_undo_cost_change" = '10.2.0.4';
alter system set "_optimizer_null_aware_antijoin" = false;
alter system set "_optimizer_extend_jppd_view_types" = false;
alter system set "_replace_virtual_columns" = false;
alter system set "_first_k_rows_dynamic_proration" = false;
alter system set "_bloom_pruning_enabled" = false;
alter system set "_optimizer_multi_level_push_pred" = false;
alter system set "_optim_peek_user_binds"=false; 
alter system set client_result_cache_size=0 scope=spfile;
alter system set result_cache_mode=MANUAL;
alter system set "_diag_hm_rc_enabled"=false; 
alter system set audit_trail=none scope=spfile;
alter system set "_memory_imm_mode_without_autosga"=false scope=both; 
alter system set "_enable_shared_pool_durations"=false scope=spfile;
alter system set deferred_segment_creation=false; 
alter system set "_datafile_write_errors_crash_instance"=false ;
alter system set "_fairness_threshold"=6 scope=spfile;
alter system set "_gc_read_mostly_locking"=false scope=spfile;
alter system set "_gc_policy_time"=0 scope=spfile;
alter system set "_gc_defer_time"=3  scope=spfile;
alter system set "parallel_force_local"=false;
alter system set "_gc_bypass_readers"=false;
alter system set "_row_cr"=false;
alter system set ddl_lock_timeout=0;
alter system set "_gby_hash_aggregation_enabled"=False scope=spfile;
alter system set "_cleanup_rollback_entries"=400 scope=spfile;
alter system set "_dbms_sql_security_level"=0                          scope=spfile;
alter system set "_bloom_pruning_enabled"=False                        scope=spfile;
alter system set "_simple_view_merging"=True                           scope=spfile;
alter system set "_enable_NUMA_optimization"=FALSE                     scope=spfile;
alter system set "_fix_control"='9344709:OFF'                           scope=spfile;
alter system set "_px_use_large_pool"=True                              scope=spfile;
alter system set "_mv_refresh_use_hash_sj"=FALSE                      scope=spfile;
alter system set "_mv_refresh_use_stats"=True                          scope=spfile;
alter system set "_like_with_bind_as_equality"=TRUE                    scope=spfile;
alter system set optimizer_secure_view_merging=false                   scope=spfile;
alter system set optimizer_capture_sql_plan_baselines=False            scope=spfile;
alter system set event="10949 TRACE NAME CONTEXT FOREVER:28401 trace name context forever, level 1"  scope=spfile;
exec  DBMS_AUTO_TASK_ADMIN.DISABLE( client_name =>  'auto optimizer stats collection', operation => NULL,window_name => NULL);
exec  DBMS_AUTO_TASK_ADMIN.DISABLE( client_name =>  'auto space advisor', operation => NULL,window_name => NULL);
exec  DBMS_AUTO_TASK_ADMIN.DISABLE( client_name => 'sql tuning advisor', operation => NULL,window_name => NULL);
commit;
exec dbms_scheduler.disable('ORACLE_OCM.MGMT_CONFIG_JOB');
exec dbms_scheduler.disable('ORACLE_OCM.MGMT_STATS_CONFIG_JOB');


// to disable histogram , you set bucket size to 1

exec DBMS_STATS.SET_PARAM( 'method_opt','FOR ALL COLUMNS SIZE 1' );
commit;



// disable 19c automatic indexing feature 

alter system set "_optimizer_auto_index_allow"=NEVER scope=spfile;
alter system set "_optimizer_use_auto_indexes"=OFF scope=spfile;

shutdown immediate;
startup ;


完成配置。

 

以上完成了基础配置,但外网客户端还是无法访问oracle服务器监听的 ,例如使用tnsping工具ping服务器监听,或sqlplus均无法登陆

 tnsping 47.XX.XX.XX

TNS Ping Utility for 64-bit Windows: Version 12.1.0.2.0 - Production on 12-9月 -2019 10:35:17

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

已使用的参数文件:
sqlnet.ora

已使用 HOSTNAME 适配器来解析别名
尝试连接 (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=47.XX.XX.XX)(PORT=1521)))
TNS-12535: TNS: 操作超时

sqlplus system/oracle@47.XX.XX.XX:1521/ORCL

SQL*Plus: Release 12.1.0.2.0 Production on 星期四 9月 12 10:39:07 2019

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

ERROR:
ORA-12170: TNS: 连接超时



我们在阿里云控制台,点选ECS服务器,更多=》网络和安全组=》安全组配置=》配置规则

 

 

点击添加安全组规则:

 

选择协议类型,ORACLE 1521 ,如果你要使用非默认ORACLE端口,即不使用1521端口,那么需要在端口范围内自行选择端口;这里我们使用默认端口,因为可以指定默认端口允许的外网IP,所以一般使用默认端口即可; 授权对象中输入 应用服务器的外网IP 。

 

 

 

点击确定后1分钟后, 可以尝试登陆:

 

 

tnsping 47.XX.XX.XX

TNS Ping Utility for 64-bit Windows: Version 12.1.0.2.0 - Production on 12-9月 -2019 10:46:46

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

已使用的参数文件:
sqlnet.ora

已使用 HOSTNAME 适配器来解析别名
尝试连接 (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=47.XX.XX.XX)(PORT=1521)))
OK (10 毫秒)


sqlplus system/oracle@47.XX.XX.XX:1521/ORCL

SQL*Plus: Release 12.1.0.2.0 Production on 星期四 9月 12 10:47:33 2019

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

上次成功登录时间: 星期四 9月  12 2019 10:33:17 +08:00

连接到:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production



可以看到登陆成功,这样我们就可以愉快的使用阿里云上的 ORACLE 19c了!!

rman generate script set until time

 
echo 'run {'  >  test2.cmd
echo 'allocate channel t1' >> test2.cmd
echo 'set until time="to_date('\'''`date "+%Y-%m-%d %H:%M:%S"`''\'','\''YYYY-MM-DD hh24:mi:ss'\'')";' >> test2.cmd
echo 'set newname for database to '+datadg';'  >> test2.cmd
echo 'restore database;' >> test2.cmd
echo 'switch  datafile all;' >> test2.cmd
echo 'recover database;' >> test2.cmd
echo 'release channel t1;' >> test2.cmd
echo '}' >> test2.cmd


关于绿盟扫描ORACLE漏洞的问题 说几点看法

之前看了这篇文章 “坑爹”的绿盟数据库漏扫,求你专业一点 ,讲几点个人的看法。

因为近期国内勒索病毒问题频发,可以观测到QQ群内大量甲方人员提问关于绿盟扫描的问题。

文章的主旨是说 绿盟的ORACLE 数据库漏洞扫描的结果比较不靠谱,已经装了的补丁,扫描的结果仍是存在漏洞,让甲方的人员很苦恼。

 

绿盟号称是在企业安全市场占有率最高的公司,拥有自己的产品和服务,涉猎企业IT环境中的几乎所有的服务器、软件的安全问题,这几年由于安全问题越来越被重视,这类安全公司确实也跟着火了。

我曾经2013年在某省移动驻场负责维护几十套Oracle数据库,每季度都会收到绿盟发来的漏洞列表,第一次发来几千项,企信部领导“高度重视”,让我连夜排查并给出解决方案。当时大多数数据库版本是11203,且安装了较新的PSU,我发现漏洞里面居然还存在2009年的CVE高危漏洞,当时就纳闷Oracle自己在2009年公布的漏洞到现在咋还没修复呢?

通过原厂确认,首先Oracle不认可任何第三方软件的漏洞扫描结果,其次绿盟的漏洞扫描机制简单粗暴,基本上没有可信度,后来和甲方DBA达成共识:1、安装最新的PSU,2、通过技术手段屏蔽绿盟的扫描。

没想到现在都2019年了,这个问题依然存在,不知道产品经理是不是去岘港度假被抓了,看看苦逼DBA们被坑害的反馈吧:

首先说说绿盟漏扫的”业余“机制吧,通过数据库版本号直接去匹配Oracle官方的CVE漏洞列表,不会检测PSU。拜托,现在大多数科技公司都在应用AI、机器学习了,您还在通过几个数字匹配表格来定义企业的信息安全分数,真是太“LOW”,哪还有安全可言。

给出了一大堆漏洞,高危的红色,让领导瑟瑟发抖,我想绿盟可能还会觉得自己很牛X吧。然而却没有给出解决方案,最多丢一个Oracle 的CVE链接给你,自己去找补丁,各个补丁之间还可能冲突,对了这里是CPU,-_-||

无力吐槽!

给乙方DBA的建议:

1、不要浪费时间尝试根据不可信的漏洞列表一一去找单个的补丁,直接安装最新的PSU即可(不过最佳实践是次新,另外12.2改为RU)。

2、觉得不好交差,或者强迫症,可以通过防火墙、端口、数据库IP限制等方式限制绿盟机器的访问;

3、把“皮球”抛给现场的绿盟GG,让他们去跟甲方解释,承认这是绿盟漏扫软件的缺陷;

4、使用专业的数据库巡检平台检查数据库,从数据库的角度去排查安全问题。

给绿盟的建议:

1、希望绿盟专业一点,牢记“专功术业”的愿景,真正做到“成就所托”,不要学早期的360通过报大量高危漏洞吓唬用户来体现自身卑微的价值;

2、学习了解Oracle的补丁策略,弄清楚One of Patch、CPU、PSU、RU的区别以及大版本小版本的关系,分别修复了哪些CVE漏洞,如果真的存在漏洞,那么给出专家的方案,不要给个CVE链接草草了事,我相信这些需求相对来说比较容易,如果需要我们可以免费支持;

3、多听听业界的声音和用户的反馈,快速迭代改进产品,不然没有核心竞争力,分分钟被替代被超越。

最后还是诚恳的呼吁绿盟团队,尽快把自己产品的缺陷修复,完善各项功能,真正为国内企业信息安全建设贡献更多的力量,“专功术业,成就所托”。

 

 

结论是让绿盟在扫描上专业一点,改进产品。

评论里又绿盟的相关人员驳斥了文章观点,认为绿盟扫描结果不准确是因为 没有做登陆扫描。

 

@楼主,看你的描述,13年就接触数据库漏洞处理并接触了漏洞扫描器,不过看起来这么多年你也没有潜心研究漏洞扫描器原理、没有思考为啥这么多年误报问题一直无法解决、也没有尝试去找厂家研究扫描功能、也没有思考国内安全厂商的技术瓶颈与困境,给出可供参考的意见和建议,通篇只有闹骚满腹,无尽抱怨,看来你这么多年也没有进步,也没有看到别人的进步,希望你还是摆正心态,踏实做事。

完全站在dba的角度去考虑安全,实话说挺无知的
1.先搞清楚基于网络的漏洞扫描原理,存活判别,应用识别,发包探测和回应到底能获取目标对象多少信息?
2.登录扫描原理,登录目标对象后到底能获取多少信息,是否会执行命令?什么情况下执行poc
3.想问问lz,哪个大厂的漏扫可以精准判断oracle漏洞?如果都不行,自己有思考过原因何在吗?光在这里瞎bb

楼主难道不知道登录扫描么?给建议,据说绿盟登录扫描很早就实现了,楼主没用过么?怪不得不知道,发文章

首先,远程扫描返回的banner不带patch,其次,还有登陆扫描的功能。
但,更多的情况是甲方不愿意把数据库的登陆方式交给除了数据库厂家外的其他乙方。吃得咸鱼抵得渴,不提供密码,扫描结果不准确那就自己受着。所以,涉密的东西还是甲方或者相关厂家自己做吧,别什么都赖给别人,自己不作为还怪别人呢。

太可怕了,lz 6年还没搞懂什么是登录扫描

 

 

上面陈述了一些问题,这里讲讲我的看法,一家之言

 

  1. 没有做登陆扫描,主要原因一般是甲方人员并不放心把账户密码给来做扫描的技术人员;以人之常情而论,我相信90%的甲方同学并不愿意做登陆扫描
  2. 因为 第一点的这个情况,导致绿盟的扫描结果,大多数情况下不准确和无价值
  3. 打一个不恰当的比方,用户去医院看病,医生让验血/拍片,但用户都不愿意验血/拍片;因为不愿意验血/拍片,所以医生几乎没有可参考的信息
  4. 在这种情况下,医生罗列了所有病人可能患的病症,列可能性嘛,多列一些总不会错
  5. 用户面对医生罗列的一大堆病症可能性,无所适从

 

这里划分一下责任:

  1. 这里可以看到,甲方的责任在于虽然有配合扫描漏洞的义务,但并不乐意真的提供账号密码来做登陆扫描。
  2. 绿盟方面可能没有强调登陆扫描的重要性, 实际上 不做登陆扫描的话 , 报告可以认为是没意义的,但这件事也就不用做了
  3. 绿盟方面因为要在没有可能获得完整信息的情况下,给出报告。强行罗列了其可以确定的可能性,这导致维护方最后要么不处理,要么很吃瘪

 

最后说几点建议:

 

  1. 如果甲方同意做漏洞扫描,就该开放账户密码给绿盟。绿盟在做扫描前应该确认这一点,如果甲方不愿意提供,那么强行去做这些事,可以说是走过场的形式主义
  2. 国内大部分非外企或上市企业使用的要么是盗版ORACLE,要么从来不购买ORACLE标准服务;这种情况下做漏洞扫描,即便出了准确结果,甲方也并没有合法途径去获得相关的安全性补丁,从网上的非官方渠道下载一些补丁,岂不是更不安全? 所以对于使用盗版ORACLE或者从不购买ORACLE标准服务的甲方,从法理上不建议去做这些漏洞扫描。这类甲方应当适当提高内网安全系数,避免ORACLE有暴露在外的可能性
  3. ORACLE每季度都会出新的PSU/SPU(以前叫CPU 即安全补丁),虽然重要度可能并不均匀(例如一年中只有一个季度发布了适应面很广,影响很大的漏洞);但可以说永远可能会发现新的安全漏洞;如果是顶级的企业,无所谓人力成本,当然可能每年升级个几次,但一般来说是不可能的,成本不可控
  4. 就Maclean的经验来看,中国和美国的顶级银行或企业都并不会如第三条这样做,打SPU的情况都是少数,除非特别需要
  5. 目前阶段,国内的主要安全趋势还是做好内网安全,能把弱密码和低版本的Windows/Linux全干掉,把无密码的MongoDB/Redis干掉,已经功德无量了
  6. 如果还有同学被绿盟的问题难住,可以耐心看下网上的许多评论,耐心和领导说明;毕竟我们纯技术人员,能做的事,或者说能做到的事并不多啊!

 

 

 

Linux内存使用的FREE命令 改进 添加available项 进一步避免FREE内存不足的误会

在各种QQ群里有大量同学会问,为什么LINUX/UNIX上free内存太少会不会有问题。

为了避免这种不必要的误会,不同的OS版本,采用了不同的策略。

例如最狠的是Solaris SUNOS ,在Solaris 8中将文件系统缓存直接算作free内存:

 

 

而Linux中从大约2014年  , 内核版本kernels 2.6.27+开始引入了MemAvailable ,其解释为:

[oracle@master ~]$ free -h
total used free shared buff/cache available
Mem: 31G 9.1G 422M 5.8G 21G 15G
Swap: 15G 11G 3.8G

available

Estimation of how much memory is available for starting new applications, without swapping. Unlike the data provided by the cache or free fields, this field takes into account page cache and also that not all re-claimable memory slabs will be reclaimed due to items being in use (MemAvailable in /proc/meminfo, available on kernels 3.14, emulated on kernels 2.6.27+, otherwise the same as free)

 

Linux Kernel 的GIT中,有比较明确的概述:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=34e431b0ae398fc54ea69ff85ec700722c9da773

/proc/meminfo: provide estimated available memory
Many load balancing and workload placing programs check /proc/meminfo to
estimate how much free memory is available.  They generally do this by
adding up "free" and "cached", which was fine ten years ago, but is
pretty much guaranteed to be wrong today.

It is wrong because Cached includes memory that is not freeable as page
cache, for example shared memory segments, tmpfs, and ramfs, and it does
not include reclaimable slab memory, which can take up a large fraction
of system memory on mostly idle systems with lots of files.

Currently, the amount of memory that is available for a new workload,
without pushing the system into swap, can be estimated from MemFree,
Active(file), Inactive(file), and SReclaimable, as well as the "low"
watermarks from /proc/zoneinfo.

However, this may change in the future, and user space really should not
be expected to know kernel internals to come up with an estimate for the
amount of free memory.

It is more convenient to provide such an estimate in /proc/meminfo.  If
things change in the future, we only have to change it in one place.

Signed-off-by: Rik van Riel <riel@redhat.com>
Reported-by: Erik Mouw <erik.mouw_2@nxp.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Diffstat
-rw-r--r--	Documentation/filesystems/proc.txt	9	
-rw-r--r--	fs/proc/meminfo.c	37	
2 files changed, 46 insertions, 0 deletions
diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt
index 22d89aa3..8533f5f 100644
--- a/Documentation/filesystems/proc.txt
+++ b/Documentation/filesystems/proc.txt
@@ -767,6 +767,7 @@ The "Locked" indicates whether the mapping is locked in memory or not.
 
 MemTotal:     16344972 kB
 MemFree:      13634064 kB
+MemAvailable: 14836172 kB
 Buffers:          3656 kB
 Cached:        1195708 kB
 SwapCached:          0 kB
@@ -799,6 +800,14 @@ AnonHugePages:   49152 kB
     MemTotal: Total usable ram (i.e. physical ram minus a few reserved
               bits and the kernel binary code)
      MemFree: The sum of LowFree+HighFree
+MemAvailable: An estimate of how much memory is available for starting new
+              applications, without swapping. Calculated from MemFree,
+              SReclaimable, the size of the file LRU lists, and the low
+              watermarks in each zone.
+              The estimate takes into account that the system needs some
+              page cache to function well, and that not all reclaimable
+              slab will be reclaimable, due to items being in use. The
+              impact of those factors will vary from system to system.
      Buffers: Relatively temporary storage for raw disk blocks
               shouldn't get tremendously large (20MB or so)
       Cached: in-memory cache for files read from the disk (the
diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c
index a77d2b2..24270ec 100644
--- a/fs/proc/meminfo.c
+++ b/fs/proc/meminfo.c
@@ -26,7 +26,11 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
 	unsigned long committed;
 	struct vmalloc_info vmi;
 	long cached;
+	long available;
+	unsigned long pagecache;
+	unsigned long wmark_low = 0;
 	unsigned long pages[NR_LRU_LISTS];
+	struct zone *zone;
 	int lru;
 
 /*
@@ -47,12 +51,44 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
 	for (lru = LRU_BASE; lru < NR_LRU_LISTS; lru++) pages[lru] = global_page_state(NR_LRU_BASE + lru); + for_each_zone(zone) + wmark_low += zone->watermark[WMARK_LOW];
+
+	/*
+	 * Estimate the amount of memory available for userspace allocations,
+	 * without causing swapping.
+	 *
+	 * Free memory cannot be taken below the low watermark, before the
+	 * system starts swapping.
+	 */
+	available = i.freeram - wmark_low;
+
+	/*
+	 * Not all the page cache can be freed, otherwise the system will
+	 * start swapping. Assume at least half of the page cache, or the
+	 * low watermark worth of cache, needs to stay.
+	 */
+	pagecache = pages[LRU_ACTIVE_FILE] + pages[LRU_INACTIVE_FILE];
+	pagecache -= min(pagecache / 2, wmark_low);
+	available += pagecache;
+
+	/*
+	 * Part of the reclaimable swap consists of items that are in use,
+	 * and cannot be freed. Cap this estimate at the low watermark.
+	 */
+	available += global_page_state(NR_SLAB_RECLAIMABLE) -
+		     min(global_page_state(NR_SLAB_RECLAIMABLE) / 2, wmark_low);
+
+	if (available < 0)
+		available = 0;
+
 	/*
 	 * Tagged format, for easy grepping and expansion.
 	 */
 	seq_printf(m,
 		"MemTotal:       %8lu kB\n"
 		"MemFree:        %8lu kB\n"
+		"MemAvailable:   %8lu kB\n"
 		"Buffers:        %8lu kB\n"
 		"Cached:         %8lu kB\n"
 		"SwapCached:     %8lu kB\n"
@@ -105,6 +141,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
 		,
 		K(i.totalram),
 		K(i.freeram),
+		K(available),
 		K(i.bufferram),
 		K(cached),
 		K(total_swapcache_pages()),


 

 

其中提到的是大量 对内存敏感的程序 是简单的将 /proc/meminfo 接口中的FREE+CACHED 的2者相加,算作可用的内存。而我们国内的同学的大多内存焦虑症,是只将FREE算作可用内存。

按照Linux Kernel Rik van Riel <riel@redhat.com>的口径是,这2种算法都是不合理的。后者是焦虑症,前者没考虑部分cache的内存未必可以被释放重用。 any load balancing and workload placing programs check /proc/meminfo to estimate how much free memory is available. They generally do this by adding up “free” and “cached”, which was fine ten years ago, but is pretty much guaranteed to be wrong today. It is wrong because Cached includes memory that is not freeable as page cache, for example shared memory segments, tmpfs, and ramfs, and it does not include reclaimable slab memory, which can take up a large fraction of system memory on mostly idle systems with lots of files.

因为在2014年,物理内存已经比较大了,更少地使用SWAP了,所以其展望未来,希望有一个接口能提供有效的评估。However, this may change in the future, and user space really should not be expected to know kernel internals to come up with an estimate for the amount of free memory. It is more convenient to provide such an estimate in /proc/meminfo. If things change in the future, we only have to change it in one place.

 

于是添加了 接口MemAvailable, 明确说明其是 评估的可以给新程序用的内存空间大小,考虑了 MemFree、SReclaimable、 the size of the file LRU lists、low watermarks in each zone。

+MemAvailable: An estimate of how much memory is available for starting new
+ applications, without swapping. Calculated from MemFree,
+ SReclaimable, the size of the file LRU lists, and the low
+ watermarks in each zone.
+ The estimate takes into account that the system needs some
+ page cache to function well, and that not all reclaimable
+ slab will be reclaimable, due to items being in use. The
+ impact of those factors will vary from system to system.

其详细算法如上文引用。

 

因为都9102年了,可以让小伙伴不要再看 free memory了, 基本上关注 available memory 就可以了!对领导也耐心解释下,看available更有意义!

 

另有同学问,为什么有free内存的情况下,会用到SWAP空间。  stackexchange上有大量相关的解释,这里引用高赞的回答:

It is normal for Linux systems to use some swap even if there is still RAM free. The Linux kernel will move to swap memory pages that are very seldom used (e.g., the getty instances when you only use X11, and some other inactive daemon).

Swap space usage becomes an issue only when there is not enough RAM available, and the kernel is forced to continuously move memory pages to swap and back to RAM, just to keep applications running. In this case, system monitor applications would show a lot of disk I/O activity.

用一句话说 就是有free memory情况下,用SWAP很正常;很少用到的内存页就会被换出去,交换导致性能不好的情况是 内存真不够了,SWAP一会换出一会换入,造成I/O开销;否则问题不大。

 

 

沪ICP备14014813号

沪公网安备 31010802001379号