Computer, Electron and Technology

google背后的分布式架构

Posted by dengwei

　　Google是与众不同的。它的独特不仅仅表现于革新的思维和充满创意的应用 (比如那个大堂里的地球模型)，更在于其有别常规的IT策略……

　　加利福尼亚州山景城(Mountain View)Google公司(Google，下称Google)总部有一个43号大楼，该建筑的中央大屏幕上显示着一个与Google地球(Google Earth)相仿的世界地图，一个转动的地球上不停地闪动着五颜六色的光点，恍如罗马宫廷的千万烛灯，每一次闪动标志着地球的这个角落一名Google用户发起了一次新的搜索。

　　这同时意味着Google又一次满足了人们对未知信息的好奇与渴望。

　　Google是与众不同的。它的独特不仅仅表现于革新的思维和充满创意的应用 (比如那个大堂里的地球模型)，更在于其有别常规的IT策略。从人们的常理来看，简单的硬件商品和免费软件是无法构建出一个帝国的，但是Google做到了。在性能调整后，Google把它们变成一个无可比拟的分布式计算平台，该平台能够支持大规模的搜索和不断涌现的新兴应用。我们原本认为这些应用都是个人消费级别的，但是Google改变了这一切。现在商业世界也在使用它们，这就令这家搜索公司显得那么与众不同。

　　GoogleWeb 服务背后的IT架构对无数使用搜索引擎的用户来说也许并不是非常重要，但它是Google几百位致力于把全球信息组织起来，实现“随处可达，随时可用”目标的工程师们的最核心工作。这就需要一个在覆盖范围和野心上都与Google的商业愿景完全相符的IT蓝图作为支撑。

　　Google 的经理们一直对公司的IT策略话题保持沉默，他们厌恶谈及特定的厂商或者产品，当被问到他们的服务器和数据中心时，他们总是闭口不谈。但与几位 Google的IT领导一起呆了一天后，我们最终得以揭示该公司的IT是如何运作的，那可不仅仅是一个运行在无数服务器集群上的、表面看来非常简单的搜索引擎。在其简单的外表下，蕴涵着许多内部研发软件、定制硬件、人工智能，以及对性能的执着追求和打破常规的人力管理模式。

　　IT理念方面，Google对同行有一条建议：尽量避免那些人人都在使用的系统和软件，以自己的方式做事会更有独特的竞争优势。

　　“企业文化决定了你的做事方式。”道格拉斯”美林(Douglas Merrill)，这位Google工程副总裁和事实上的首席信息官(CIO) 指出，“到了我们这样的发展阶段，企业观念和文化非常与众不同，这也反过来鞭策我们必须要采用与众不同的方式来运行那些他人看来很常规的系统。”
　　Google 最大的IT优势在于它能建造出既富于性价比(并非廉价)又能承受极高负载的高性能系统。因此IT顾问史蒂芬”阿诺德(Stephen Arnold)指出，Google与竞争对手，如亚马逊网站(Amazon)、电子港湾公司(eBay)、微软公司(Microsoft，下称微软)和雅虎公司 (Yahoo，下称雅虎)等公司相比，具有更大的成本优势。Google程序员的效率比其他Web公司同行们高出50%～100%，原因是Google已经开发出了一整套专用于支持大规模并行系统编程的定制软件库。据他估算，其他竞争公司可能要花上四倍的时间才能获得同等的效果。

　　打造服务器

　　Google 究竟是怎样做到这点的呢?其中一个手段，美林认为，“是因为我们自己动手打造硬件。”Google并不制造计算机系统，但它根据自己的参数定制硬件，然后像MTV的节目“靓车打造”(Pimp My Ride)那样自己安装和调整硬件系统。开源程序经理克里斯”迪博纳(Chris DiBona)评论道：“我们很善于购买商业服务器，并且改造他们为我们所用，最后把性能压榨和发挥到极致，以致有时候他们热得像要融化了似的。”

　　这种亲手打造的方式，来源于Google从车库诞生时与生俱来的节俭风格，更与Google那超大型的系统规模息息相关，良好的习惯一直延续至今。据说 Google在65个数据中心拥有20万～45万台服务器—这个数目会有偏差(取决于你如何定义服务器和由谁来做这项统计)。但是，不变的是持续上升的趋势。

　　Google不会去讨论这些资产，因为它认为保密也是一种竞争优势。事实上，Google之所以喜欢开源软件也是因为它的私密性。“如果我们购买了软件许可或代码许可，人们只要对号入座，就可以猜出Google的IT基础架构。”迪博纳分析说， “使用开源软件，就使我们多了一条把握自己命运的途径。”

　　Google喜欢规模化的服务器运行方式。当有成百上千台机器时，定制服务器的优势也会成倍增加，效果也会更趋明显。Google正在俄勒冈州哥伦比亚河边的达勒斯市建造一个占地30亩的数据中心，在那儿它可以获得运算和降温需要的低价水力电力能源(参见边栏《Google数据中心自有一套》)。

　　Google以“单元”(Cell)的形式组织这些运行 Linux操作系统的服务器，迪博纳把这种形式比喻成互联网服务的“磁盘驱动器”(但别和一直谣传的Google存储服务Gdrive混淆了，“并没有 Gdrive这回事。”一位Google女发言人明确表示。)，公司的软件程序都驻扎在这些并不昂贵的电脑机箱里，由程序员决定它们的冗余工作量。这种由很多单元组成的文件系统代替了商业存储设备;迪博纳表示Google这些单元设备更易于建造和维护，他还暗示他们能处理更大规模的数据。

　　Google 不会漏过对任何技术细节的关注。多年来，公司的工程师就在研究微处理器的内部工作机制，随着Google规模的持续壮大，必然会用到特别定制和调节过的芯片。知名工程师路易斯”巴罗索(Luiz Barroso)去年在一篇发表在工业杂志上的论文中证实，近年来Google的主要负荷都由单核设计的系统承担着。但许多服务器端的应用，如 Google搜索索引服务，所需的并行计算在单核芯片的指令级别上执行得并不好。

　　曾在数据设备公司(Digital Equipment)和康柏公司(Compaq)当过芯片设计师的巴罗索认为，随着AMD公司、英特尔公司(Intel)、太阳计算机系统公司(Sun)开始制造多核芯片，必将会出现越来越多芯片级别的并行计算。

　　Google 也曾考虑过自己制造计算机芯片，但从业界潮流来看，这个冒险的举动似乎不是很必要。“微处理器的设计非常复杂而且成本昂贵，”运营高级副总裁乌尔斯”霍尔茨勒(Urs Holzle)表示。Google宁愿与芯片制造商合作，让他们去理解自己的应用并设计适合的芯片。这是一种客户建议式的设计，其关注点在于总体吞吐量、效能，以及耗电比，而不是看单线程的峰值性能。霍尔茨勒表示，“这也是最近多核CPU的设计潮流与未来方向。”
裁缝般地定制软件

　　为了能尽量压榨硬件性能，Google开发了相当数量的定制软件。创新产品主要包括用于简化处理和创建大规模数据集的编程模型 MapReduce;用于存储和管理大规模数据的系统BigTable;分析分布式运算环境中大规模数据集的解释编程语言Sawzall;用于数据密集型应用的分布式文件系统的 “Google文件系统”(Google File System);还有为处理分布式系统队列分组和任务调度的“Google工作队列”(Google Workqueue)。

　　正是从Sawzall这些工具里体现出Google对计算效率的执著关注。并不是每家公司都能从底层去解决效率问题，但是对Google来说，为常规关系型数据库无法容纳的大规模数据集专门设计一种编程语言是完全合理的。即使其他编程工具可以解决问题，Google的工程师们仍然会为了追求效率而另外开发一套定制方案。Google工程师认为，Sawzall能与C++中的MapReduce相媲美，而且它更容易编写一些。

　　Google 对效率的关注使它不可能对标准Linux内核感到满意;Google会根据自己的需要运行修改过的内核版本。通过调整Linux的底层性能，Google 工程师们在提高了整体系统可靠性的基础上，还一并解决了数据损坏和数据瓶颈等一系列棘手问题。对内核的修改也使Google的计算机集群系统因为通信效率的提高而运行得更快。

　　当然，Google偶尔也会出现系统故障，情况一旦发生，无数的用户就会受到影响了。三年前一次持续30分钟的系统故障使20%的搜索流量受到影响。

　　Google 开发了自己的网站服务器却没有使用开源的Apache服务器，尽管它在网站服务器的市场占有率超过60%。迪博纳认为，Google的网站服务器可以运行在更多数量的主机上，对Google站点上内容庞大又彼此互相依赖的应用程序来说，这种服务器的负载均衡能力远比Apache的能力更高。同时，在用标准公共网关接口(CGI)访问数据库动态网页方面，Google服务器的编程难度要比 Apache更高，但是最终运行速度却更快。“如果我们能够压榨出10%～20%的性能，我们就可以节省出更多系统资源、电量和人力了。”迪博纳在总结中指出。

　　Google还设计了自己的客户关系管理(CRM)系统用于支持自己基于竞价和点击的互联网广告收费业务。但对是否需要设计自己的工具，Google的态度也不是一成不变的。比如在财会软件上，它就使用了甲骨文公司(Oracle)的Financials软件。

　　美林拿着一只叉子举例说明现成的产品也可以带来价值。但在有些场合现成的软件产品就不一定适用了。“我们的文化在各个层面对我们的运作都有深远影响，”他表示，“所以我们不想让购买所得的工具改变我们的工作方式和文化层面。”
Google’s BigTable 原理（翻译）

题记：google 的成功除了一个个出色的创意外，还因为有 Jeff Dean 这样的软件架构天才。
—— 编者

官方的 Google Reader blog 中有对BigTable 的解释。这是Google 内部开发的一个用来处理大数据量的系统。这种系统适合处理半结构化的数据比如 RSS 数据源。以下发言是 Andrew Hitchcock 在 2005 年10月18号基于： Google 的工程师 Jeff Dean 在华盛顿大学的一次谈话 (Creative Commons License).

首先，BigTable 从 2004 年初就开始研发了，到现在为止已经用了将近8个月。（2005年2月）目前大概有100个左右的服务使用BigTable，比如： Print,Search History,Maps和 Orkut。根据Google的一贯做法，内部开发的BigTable是为跑在廉价的PC机上设计的。BigTable 让Google在提供新服务时的运行成本降低，最大限度地利用了计算能力。

BigTable 是建立在 GFS ，Scheduler ，Lock Service 和 MapReduce 之上的。

每个Table都是一个多维的稀疏图 sparse map。Table 由行和列组成，并且每个存储单元 cell 都有一个时间戳。在不同的时间对同一个存储单元cell有多份拷贝，这样就可以记录数据的变动情况。在他的例子中，行是URLs ，列可以定义一个名字，比如：contents。Contents 字段就可以存储文件的数据。或者列名是：”language”，可以存储一个“EN”的语言代码字符串。

为了管理巨大的Table，把Table根据行分割，这些分割后的数据统称为：Tablets。每个Tablets大概有 100-200 MB，每个机器存储100个左右的 Tablets。底层的架构是：GFS。由于GFS是一种分布式的文件系统，采用Tablets的机制后，可以获得很好的负载均衡。比如：可以把经常响应的表移动到其他空闲机器上，然后快速重建。

Tablets在系统中的存储方式是不可修改的 immutable 的SSTables，一台机器一个日志文件。当系统的内存满后，系统会压缩一些Tablets。由于Jeff在论述这点的时候说的很快，所以我没有时间把听到的都记录下来，因此下面是一个大概的说明：

压缩分为：主要和次要的两部分。次要的压缩仅仅包括几个Tablets，而主要的压缩时关于整个系统的压缩。主压缩有回收硬盘空间的功能。Tablets的位置实际上是存储在几个特殊的BigTable的存储单元cell中。看起来这是一个三层的系统。
客户端有一个指向METAO的Tablets的指针。如果METAO的Tablets被频繁使用，那个这台机器就会放弃其他的tablets专门支持 METAO这个Tablets。METAO tablets 保持着所有的META1的tablets的记录。这些tablets中包含着查找tablets的实际位置。（老实说翻译到这里，我也不太明白。）在这个系统中不存在大的瓶颈，因为被频繁调用的数据已经被提前获得并进行了缓存。

现在我们返回到对列的说明：列是类似下面的形式： family:optional_qualifier。在他的例子中，行：www.search-analysis.com 也许有列：”contents:其中包含html页面的代码。 “ anchor:cnn.com/news” 中包含着相对应的url，”anchor:www.search-analysis.com/” 包含着链接的文字部分。列中包含着类型信息。
(翻译到这里我要插一句，以前我看过一个关于万能数据库的文章，当时很激动，就联系了作者，现在回想起来，或许google的 bigtable 才是更好的方案，切不说分布式的特性，就是这种建华的表结构就很有用处。)

注意这里说的是列信息，而不是列类型。列的信息是如下信息，一般是：属性/规则。比如：保存n份数据的拷贝或者保存数据n天长等等。当 tablets 重新建立的时候，就运用上面的规则，剔出不符合条件的记录。由于设计上的原因，列本身的创建是很容易的，但是跟列相关的功能确实非常复杂的，比如上文提到的类型和规则信息等。为了优化读取速度，列的功能被分割然后以组的方式存储在所建索引的机器上。这些被分割后的组作用于列 ,然后被分割成不同的 SSTables。这种方式可以提高系统的性能，因为小的，频繁读取的列可以被单独存储，和那些大的不经常访问的列隔离开来。

在一台机器上的所有的 tablets 共享一个log，在一个包含1亿的tablets的集群中，这将会导致非常多的文件被打开和写操作。新的log块经常被创建，一般是64M大小，这个GFS的块大小相等。当一个机器down掉后，控制机器就会重新发布他的log块到其他机器上继续进行处理。这台机器重建tablets然后询问控制机器处理结构的存储位置，然后直接对重建后的数据进行处理。这个系统中有很多冗余数据，因此在系统中大量使用了压缩技术。

Dean 对压缩的部分说的很快，我没有完全记下来，所以我还是说个大概吧：压缩前先寻找相似的 \行，列，和时间数据。

他们使用不同版本的： BMDiff 和 Zippy 技术。

BMDiff 提供给他们非常快的写速度： 100MB/s – 1000MB/s 。Zippy 是和 LZW 类似的。Zippy 并不像 LZW 或者 gzip 那样压缩比高，但是他处理速度非常快。

Dean 还给了一个关于压缩 web 蜘蛛数据的例子。这个例子的蜘蛛包含 2.1B 的页面，行按照以下的方式命名：“com.cnn.www/index.html:http”.在未压缩前的web page 页面大小是：45.1 TB ，压缩后的大小是：4.2 TB ，只是原来的 9.2%。Links 数据压缩到原来的 13.9% , 链接文本数据压缩到原来的 12.7%。
Read the rest of this entry »

PHP:set_error_handler ……need more

Posted by dengwei

本来想自己写个 error 处理的 logger 结果发现通过 set_error_handler 没办法捕获到 fatal error & parse error 唉，真愁人呐。

在 php.net 上也没有找到办法，后来反到是在 zend.com 上找到了解决 catch fatal error 的办法就是在 auto_prepend_file 和 auto_append_file 上做手脚。

prepend 的文件里面有一个 string 里面是个 error page 的 html 包括一个 script 可以把错误信息发送到 server 的一个 api 上。

而在 append 的文件里通过 ob_get_contents() 来把那个 string 给去掉，如果能去掉就说明程序中间的执行流程正确，无错，如果有 fatal error 则没有办法到达 append 文件这步，所以会显示那个 html 页面。

方法还可以，但是因为每个 php 都要 prepend & append 可能效率会不行，和 Xuanyan 讨论了一小会暂时还是没有什么好办法来捕获 fatal & parse error 。

XCache & XDebug on road

Posted by gavinkwoe

终于配置上 XCache 和 XDebug 了，可惜的是 php-java-bridge 一直没搞好，只有双击运行 JavaBridge 后才行，唉，要是能内置到 PHP 里就好了。

继续研究 FastCGI & Lighttpd

如果说之前在 UUSee 是向上研究，既“抽象”、“架构”的话，那么来 IMobile 之后研究方向则是向下，研究底层，研究以前没注意到的更细节的地方了。

Good days, good luck.

redhad下的openssl(安装和卸载)

Posted by dengwei

转至: http://blog.csdn.net/baitianhai/archive/2004/10/27/155461.aspx

最近在鼓捣redhat linux，想自己以源代码方式安装软件，不想用rpm方式安装。

首先从httpd开始，先卸载在安装倒是比较容易，不过后来像添加ssl功能，发现编译的时候需要用openssl的安装目录，本人比较愚笨，一顿好找也没有找到，于是就想把openssl也以源代码方式安装。先卸载，此时出现问题，系统好多东西依赖于openssl的库，我查了好多资料也没找到什么办法，于是我最后一狠心，用rpm -e –nodeps给卸载了，然后手动安装了openssl，然后重新启动，这下坏了，好多服务都起不来了，smb,ssh等等，图形模式也起不来了，我欲哭无泪。

因为我是在虚拟机上安装的，smb起不来了，我只能重新安装系统了。这次安装我大多数东西都没选择，一路安装完毕，结果在文本方式发现vi编辑没有颜色了，哎，也不知道是少装了那个东西弄得（各位谁知道麻烦告诉告诉我一下），只能按照猜测重新安装了又添加了一些东西。不过幸运的vi高亮显示功能又有了，遗憾的是具体是那个软件我还是不清楚。有了上次的教训我不敢轻易卸掉系统原来的openssl了，我从网上搜索到了一篇安装openssl的英文文章，地址在 http://www.devside.net/web/server/linux/openssl 我按照上面说的安装了zlib,openssl。步骤简介如下（怕以后忘了）

安装zlib

Home : http://www.gzip.org/zlib/

Package(linux source) : http://www.gzip.org/zlib/

Our Configuration

Install to : /usr/local

Module types : dynamically and staticly loaded modules, *.so and *.a

Build Instructions

zlib library files are placed into /usr/local/lib and zlib header files are placed into /usr/local/include, by default.

Build static libraries

…/zlib-1.2.1]# ./configure

…/zlib-1.2.1]# make test

…/zlib-1.2.1]# make install

Build shared libraries

…/zlib-1.2.1]# make clean

…/zlib-1.2.1]# ./configure –shared

…/zlib-1.2.1]# make test

…/zlib-1.2.1]# make install

…/zlib-1.2.1]# cp zutil.h /usr/local/include

…/zlib-1.2.1]# cp zutil.c /usr/local/include

/usr/local/lib should now contain…

libz.a

libz.so -> libz.so.1.2.1

libz.so.1 -> libz.so.1.2.1

libz.so.1.2.1

/usr/local/include should now contain…

zconf.h

zlib.h

zutil.h

[Optional] Instructions for non-standard placement of zlib

Create the directory that will contain zlib

…/zlib-1.2.1]# mkdir /usr/local/zlib

Follow the given procedure above, except

…/zlib-1.2.1]# ./configure –prefix=/usr/local/zlib

Update the Run-Time Linker

/etc/ld.so.cache will need to be updated with the new zlib shared lib: libz.so.1.2.1

For standard zlib installation…

Add /usr/local/lib to /etc/ld.so.conf, if specified path is not present

/etc]# ldconfig

If zlib was installed with a prefix…

Add /usr/local/zlib/lib to /etc/ld.so.conf

/etc]# ldconfig

安装openssl

Download

Home : http://www.openssl.org/

Package(source) : openssl-0.9.7d.tar.gz

Our Configuration

install to : /usr/local/ssl

module types : dynamically and staticly loaded modules, *.so *.a

Build Instructions

…/openssl-0.9.7d]# ./config

–prefix=/usr/local/ssl

[default location]

shared

[in addition to the usual static libraries, create shared libraries]

zlib-dynamic

[like "zlib", but has OpenSSL load the zlib library dynamically when needed]

…/openssl-0.9.7d]# ./config -t

[display guess on system made by ./config]

…/openssl-0.9.7d]# make

…/openssl-0.9.7d]# make test

…/openssl-0.9.7d]# make install

Update the Run-time Linker

ld.so.cache will need to be updated with the location of the new OpenSSL shared libs: libcrypto.so.0.9.7 and libssl.so.0.9.7

Sometimes it is sufficient to just add these two files to /lib, but we recommend you follow these instructions instead.

Edit /etc/ld.so.conf

Add /usr/local/ssl/lib to the bottom.

…]# ldconfig

Update the PATH

Edit /root/.bash_profile

Add /usr/local/ssl/bin to the PATH variable.

Re-login

Testing

…]# openssl version

Should display OpenSSL 0.9.7d 17 Mar 2004

If an older version is shown, your system contains a previously installed OpenSSL.

Repeate the steps in Update the PATH, except place the specified location at the start of the PATH variable.

[the older openssl, on most systems, is located under /usr/bin]

[the command 'which openssl' should display the path of the openssl that your system is using]

/usr/local/ssl/bin]# ./openssl version should display the correct version.

但是我最后没有得到想要的结果，系统原来的openssl还是没能卸载掉，我该怎么做那？我继续搜索资料，哈，幸运的我找了，在一个国内论坛上是这么说的

cd /usr/local/ssl/lib

ln -s libcrypto.so.0.9.7 libcrypto.so.2

ln -s libssl.so.0.9.7 libssl.so.2

//最后要刷新系统的动态连接库配置

echo /usr/local/ssl/lib >> /etc/ld.so.conf

ldconfig -v

这下子我豁然开朗，原来依赖的那2个文件是个软链接啊，我把它修改为我现在真正的openssl库文件不是就行了吗？于是一顿忙碌后，我终于执行了 rpm -e -nodeps ，然后重新启动系统，一路运行下去，全是绿灯。一时间感觉自己好幸福啊

为了这个问题我查了国内的几个比较大的unix/linux网站都没找到资料，不过从这里http://bbs.netbuddy.org/unix/737.html还是找到了（国外的E文大概意思能看懂，但是查找起来还是没找到，也不知道这方面好点的网站），

OpenSSL相关命令（for Linux）详细介绍

Posted by dengwei

转至: http://blog.ixpub.net/8400463

加密算法：

对称加密算法：

DES、IDEA、RC2、RC4、AES、Skipjack ……

非对称加密算法：

RSA、DSA、DiffieHellman、PKCS、PGP ……

单向的HASH算法属于报文摘要算法，虽然有些也出自OpenSSL库。
命令操作：

1、生成普通私钥：
[weigw@TEST src]$ openssl genrsa -out privatekey.key 1024

Generating RSA private key, 1024 bit long modulus ….++++++ …….++++++ e is 65537 (0×10001)

2、生成带加密口令的密钥：

[weigw@TEST src]$ openssl genrsa -des3 -out privatekey.key 1024

Generating RSA private key, 1024 bit long modulus …………++++++ …………………++++++ e is 65537 (0×10001) Enter pass phrase for privatekey.key: Verifying - Enter pass phrase for privatekey.key:

在生成带加密口令的密钥时需要自己去输入密码。对于为密钥加密现在提供了一下几种算法：
-des encrypt the generated key with DES in cbc mode

-des3 encrypt the generated key with DES in ede cbc mode (168 bit key)

-aes128, -aes192, -aes256 encrypt PEM output with cbc aes

去除密钥的口令：
[weigw@TEST src]$ openssl rsa -in privatekey.key -out

privatekey.key Enter pass phrase for privatekey.key: writing RSA key

通过生成的私钥去生成证书：

[weigw@TEST src]$ openssl req -new -x509 -key privatekey.key -out cacert.crt -days 1095

You are about to be asked to enter information that will be incorporated into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter ‘.’, the field will be left blank.

—–

Country Name (2 letter code) [GB]:CN

State or Province Name (full name) [Berkshire]:beijing

Locality Name (eg, city) [Newbury]:beijing

Organization Name (eg, company) [My Company Ltd]:wondersoft

Organizational Unit Name (eg, section) []:develop

Common Name (eg, your name or your server’s hostname) []:WeiGW

Email Address []:weigongwan@sina.com

在生成证书的时候需要按照提示输入一些个人信息。

通过私钥生成公钥：

[weigw@TEST src]$ openssl rsa -in privatekey.key -pubout -out pubkey.key writing RSA key

格式转换：（证书、私钥、公钥）（PEM <—–>DER）

[weigw@TEST src]$ openssl x509 -in cacert.crt -inform. PEM -out cacert.der -outform. DER

[weigw@TEST src]$

[weigw@TEST src]$ openssl rsa -in privatekey.key -inform. PEM -out privatekey.der -outform. DER

writing RSA key

[weigw@TEST src]$ openssl rsa -pubin -in pubkey.key -inform. PEM -pubout -out pubkey.der -outform. DER

writing RSA key

从DER格式转换成PEM格式一样，就是把inform的格式改成DERoutform的格式改成PEM即可。

下面是一个服务器和客户端认证的证书、私钥生成方法：（server.crt、client.crt、ca.crt）

第一步：生成私钥

[weigw@TEST bin]$ openssl genrsa -out server.key 1024

Generating RSA private key, 1024 bit long modulus .++++++ ..
………++++++ e is 65537 (0×10001)

[weigw@TEST bin]$ openssl genrsa -out client.key 1024

Generating RSA private key, 1024 bit long modulus …++++++ ……
……….++++++ e is 65537 (0×10001)

[weigw@TEST bin]$ openssl genrsa -out ca.key 1024

Generating RSA private key, 1024 bit long modulus …….
..++++++ ………++++++ e is 65537 (0×10001)

[weigw@TEST bin]$

第三步：申请证书（为请求文件签名）

[weigw@TEST bin]$ openssl ca -in server.csr -out server.crt -cert ca.crt -keyfile ca.key

[weigw@TEST bin]$ openssl ca -in client.csr -out client.crt -cert ca.crt -keyfile ca.key

如果在这步出现错误信息：

[weigw@TEST bin]$ openssl ca -in client.csr -out client.crt -cert ca.crt -keyfile ca.key

Using configuration from /usr/share/ssl/openssl.cnf I am unable to access the ./demoCA/newcerts directory ./demoCA/newcerts: No such file or directory

[weigw@TEST bin]$

自己手动创建一个CA目录结构：
[weigw@TEST bin]$ mkdir ./demoCA
[weigw@TEST bin]$ mkdir demoCA/newcerts
创建个空文件：
[weigw@TEST bin]$ vi demoCA/index.txt
向文件中写入01：
[weigw@TEST bin]$ vi demoCA/serial

合并证书文件（crt）和私钥文件（key）：

[weigw@TEST bin]$ cat client.crt client.key > client.pem [weigw@TEST bin]$ cat server.crt server.key > server.pem

合并成pfx证书：

[weigw@TEST bin]$ openssl pkcs12 -export -clcerts -in client.crt -inkey client.key -out client.p12

Enter Export Password:

Verifying - Enter Export Password:

[weigw@TEST bin]$openssl pkcs12 -export -clcerts -in server.crt -inkey server.key -out server.p12
Enter Export Password:
Verifying - Enter Export Password:

文本化证书：

[weigw@TEST bin]$ openssl pkcs12 -in client.p12 -out client.txt Enter Import Password:

MAC verified OK

Enter PEM pass phrase: Verifying - Enter PEM pass phrase:

[weigw@TEST bin]$openssl pkcs12 -in server.p12 -out server.txt

Enter Import Password:

MAC verified OK

Enter PEM pass phrase: Verifying - Enter PEM pass phrase:

屏幕模式显式：（证书、私钥、公钥）

[weigw@TEST bin]$ openssl x509 -in client.crt -noout -text -modulus

[weigw@TEST bin]$ openssl rsa -in server.key -noout -text -modulus

[weigw@TEST bin]$ openssl rsa -in server.pub -noout -text -modulus

得到DH：

[weigw@TEST bin]$ openssl dhparam -out dh1024.pem 1024

[收藏] Asian NTP Server pool

Posted by dengwei

Philippines — ph.pool.ntp.org (1)
Malaysia — my.pool.ntp.org (4)
Turkey — tr.pool.ntp.org (1)
Singapore — sg.pool.ntp.org (2)
India — in.pool.ntp.org (1)
Hong Kong — hk.pool.ntp.org (1)
United Arab Emirates — ae.pool.ntp.org (0)
Japan — jp.pool.ntp.org (6)
Bangladesh — bd.pool.ntp.org (0)
Israel — il.pool.ntp.org (3)
Korea — kr.pool.ntp.org (4)
Thailand — th.pool.ntp.org (1)
Iran — ir.pool.ntp.org (0)
Taiwan — tw.pool.ntp.org (15)
China — cn.pool.ntp.org (6)
Indonesia — id.pool.ntp.org (3)

[超长篇] Inject Your Code to a Portable Executable File

Posted by dengwei

转至: http://www.codeguru.com/cpp/w-p/system/misc/article.php/c11393

Downloads

test1.zip -

Windows NT 3.51 (I mean, Win3.1, Win95, Win98 were not perfect OSs). The MS-DOS data causes that your executable file to have the performance inside MS-DOS and the MS-DOS Stub program lets it display: "This program can not be run in MS-DOS mode" or "This program can be run only in Windows mode", or some things like these comments when you try to run a Windows EXE file inside MS-DOS 6.0, where there is no footstep of Windows. Thus, this data is reserved for the code to indicate these comments in the MS-DOS operating system. The most interesting part of the MS-DOS data is "MZ"! Can you believe, it refers to the name of "Mark Zbikowski", one of the first Microsoft programmers?

0 Preface

You might demand to comprehend the ways a virus program injects its procedure into the interior of a portable executable file and corrupts it, or you are interested in implementing a packer or a protector to encrypt the data of your portable executable (PE) file. This article is committed to represent a brief discussion to realize the performance that is accomplished by EXE tools or some kinds of mal-ware.

You can employ this article’s source code to create your custom EXE builder. It could be used to make an EXE protector in the right way, or with the wrong intention, to spread a virus. However, my purpose of writing this article has been the first application, so I will not be responsible for the immoral usage of these methods.

1 Prerequisites

There are no specific mandatory prerequisites to follow the topics in this article. If you are familiar with a debugger and also the portable file format, I suggest you to drop to Sections 2 and 3; the whole of these sections has been made for people who don’t have any knowledge regarding the EXE file format or debuggers.

2 Portable Executable File Format

The Portable Executable file format was defined to provide the best way for the Windows Operating System to execute code and also to store the essential data that is needed to run a program—for example constant data, variable data, import library links, and resource data. It consists of MS-DOS file information, Windows NT file information, Section Headers, and Section images, as shown in Table 1.

2.1 The MS-DOS data

These data let you remember the first days of developing the Windows Operating System. You were at the beginning of a way to achieve a complete Operating System such as

To me, only the offset of the PE signature in the MS-DOS data is important, so I can use it to find the position of the Windows NT data. I just recommend that you take a look at Table 1, and then observe the structure of IMAGE_DOS_HEADER in the <winnt.h> header in the <Microsoft Visual Studio .net path>\VC7\PlatformSDK\include\ folder or the <Microsoft Visual Studio 6.0 path>\VC98\include\ folder. I do not know why the Microsoft team has forgotten to provide some comment about this structure in the MSDN library!

typedef struct _IMAGE_DOS_HEADER { // DOS .EXE header "MZ"    WORD   e_magic;                // Magic number    WORD   e_cblp;                 // Bytes on last page of file    WORD   e_cp;                   // Pages in file    WORD   e_crlc;                 // Relocations    WORD   e_cparhdr;              // Size of header in                                   // paragraphs    WORD   e_minalloc;             // Minimum extra paragraphs                                   // needed    WORD   e_maxalloc;             // Maximum extra paragraphs                                   // needed    WORD   e_ss;                   // Initial (relative) SS                                   // value    WORD   e_sp;                   // Initial SP value    WORD   e_csum;                 // Checksum    WORD   e_ip;                   // Initial IP value    WORD   e_cs;                   // Initial (relative) CS                                   // value    WORD   e_lfarlc;               // File address of relocation                                   // table    WORD   e_ovno;                 // Overlay number    WORD   e_res[4];               // Reserved words    WORD   e_oemid;                // OEM identifier                                   // (for e_oeminfo)    WORD   e_oeminfo;              // OEM information;                                   // e_oemid specific    WORD   e_res2[10];             // Reserved words    LONG   e_lfanew;               // File address of the new                                   // exe header  } IMAGE_DOS_HEADER, *PIMAGE_DOS_HEADER;

e_lfanew is the offset that refers to the position of the Windows NT data. I have provided a program to obtain the header information from an EXE file and to display it to you. To use the program, just try:

PE Viewer

(Full Size Image)

This sample is useful for the whole of this article.

Table 1: Portable Executable file format structure

MS-DOS information	`IMAGE_DOS_ HEADER`	DOS EXE Signature	00000000 ASCII "MZ"00000002 DW 009000000004 DW 000300000006 DW 000000000008 DW 00040000000A DW 00000000000C DW FFFF0000000E DW 000000000010 DW 00B800000012 DW 000000000014 DW 000000000016 DW 000000000018 DW 00400000001A DW 00000000001C DB 00b&b&0000003B DB 000000003C DD 000000F0
		`DOS_PartPag`
		`DOS_PageCnt`
		`DOS_ReloCnt`
		`DOS_HdrSize`
		`DOS_MinMem`
		`DOS_MaxMem`
		`DOS_ReloSS`
		`DOS_ExeSP`
		`DOS_ChkSum`
		`DOS_ExeIPP`
		`DOS_ReloCS`
		`DOS_TablOff`
		`DOS_Overlay`
		`b&`Reserved words`b&`
		Offset to PE signature
	MS-DOS Stub Program	00000040 ..B:..B4.C!B8\LC!This program canno00000060 t be run in DOS mode....$.......
Windows NT information `IMAGE_ NT_HEADERS`	Signature	PE signature (PE)	000000F0 ASCII "PE"
	`IMAGE_ FILE_HEADER`	`Machine`	000000F4 DW 014C000000F6 DW 0003000000F8 DD 3B7D8410000000FC DD 0000000000000100 DD 0000000000000104 DW 00E000000106 DW 010F
		`NumberOfSections`
		`TimeDateStamp`
		`PointerToSymbolTable`
		`NumberOfSymbols`
		`SizeOfOptionalHeader`
		`Characteristics`
	`IMAGE_ OPTIONAL_ HEADER32`	`MagicNumber`	00000108 DW 010B0000010A DB 070000010B DB 000000010C DD 0001280000000110 DD 00009C0000000114 DD 0000000000000118 DD 000124750000011C DD 0000100000000120 DD 0001400000000124 DD 0100000000000128 DD 000010000000012C DD 0000020000000130 DW 000500000132 DW 000100000134 DW 000500000136 DW 000100000138 DW 00040000013A DW 00000000013C DD 0000000000000140 DD 0001F00000000144 DD 0000040000000148 DD 0001D7FC0000014C DW 00020000014E DW 800000000150 DD 0004000000000154 DD 0000100000000158 DD 001000000000015C DD 0000100000000160 DD 0000000000000164 DD 00000010
		`MajorLinkerVersion`
		`MinorLinkerVersion`
		`SizeOfCode`
		`SizeOfInitializedData`
		`SizeOfUninitializedData`
		`AddressOfEntryPoint`
		`BaseOfCode`
		`BaseOfData`
		`ImageBase`
		`SectionAlignment`
		`FileAlignment`
		`MajorOSVersion`
		`MinorOSVersion`
		`MajorImageVersion`
		`MinorImageVersion`
		`MajorSubsystemVersion`
		`MinorSubsystemVersion`
		`Reserved`
		`SizeOfImage`
		`SizeOfHeaders`
		`CheckSum`
		`Subsystem`
		`DLLCharacteristics`
		`SizeOfStackReserve`
		`SizeOfStackCommit`
		`SizeOfHeapReserve`
		`SizeOfHeapCommit`
		`LoaderFlags`
		`NumberOfRvaAndSizes`
		`IMAGE_ DATA_DIRECTORY[16]`	Export Table
			Import Table
			Resource Table
			Exception Table
			Certificate File
			Relocation Table
			Debug Data
			Architecture Data
			Global Ptr
			TLS Table
			Load Config Table
			Bound Import Table
			Import Address Table
			Delay Import Descriptor
			COM+ Runtime Header
			Reserved
Sections information	`IMAGE_ SECTION_ HEADER[0]`	`Name[8]`	000001E8 ASCII".text"000001F0 DD 000126B0000001F4 DD 00001000000001F8 DD 00012800000001FC DD 0000040000000200 DD 0000000000000204 DD 0000000000000208 DW 00000000020A DW 00000000020C DD 60000020 CODE\|EXECUTE\|READ
		`VirtualSize`
		`VirtualAddress`
		`SizeOfRawData`
		`PointerToRawData`
		`PointerToRelocations`
		`PointerToLineNumbers`
		`NumberOfRelocations`
		`NumberOfLineNumbers`
		`Characteristics`
	`b& b& b& IMAGE_ SECTION_ HEADER[n]`	00000210 ASCII".data"; SECTION00000218 DD 0000101C ; VirtualSize = 0x101C0000021C DD 00014000 ; VirtualAddress = 0x1400000000220 DD 00000A00 ; SizeOfRawData = 0xA0000000224 DD 00012C00 ; PointerToRawData = 0x12C0000000228 DD 00000000 ; PointerToRelocations = 0x00000022C DD 00000000 ; PointerToLineNumbers = 0x000000230 DW 0000 ; NumberOfRelocations = 0x000000232 DW 0000 ; NumberOfLineNumbers = 0x000000234 DD C0000040 ; Characteristics = INITIALIZED_DATA\|READ\|WRITE00000238 ASCII".rsrc"; SECTION00000240 DD 00008960 ; VirtualSize = 0x896000000244 DD 00016000 ; VirtualAddress = 0x1600000000248 DD 00008A00 ; SizeOfRawData = 0x8A000000024C DD 00013600 ; PointerToRawData = 0x1360000000250 DD 00000000 ; PointerToRelocations = 0x000000254 DD 00000000 ; PointerToLineNumbers = 0x000000258 DW 0000 ; NumberOfRelocations = 0x00000025A DW 0000 ; NumberOfLineNumbers = 0x00000025C DD 40000040 ; Characteristics = INITIALIZED_DATA\|READ
	`SECTION[0]`	00000400 EA 22 DD 77 D7 23 DD 77 C"C.wC.#C.w00000408 9A 18 DD 77 00 00 00 00 E!.C.w....00000410 2E 1E C7 77 83 1D C7 77 ..C.wF..C.w00000418 FF 1E C7 77 00 00 00 00 C?.C.w....00000420 93 9F E7 77 D8 05 E8 77 b.E8C'wC..C(w00000428 FD A5 E7 77 AD A9 E9 77 C=B%C'wB)C)w00000430 A3 36 E7 77 03 38 E7 77 B#6C'w.8C'w00000438 41 E3 E6 77 60 8D E7 77 AC#C&w`BC'w00000440 E6 1B E6 77 2B 2A E7 77 C&.C&w+C'w00000448 7A 17 E6 77 79 C8 E6 77 z.C&wyC.C&w00000450 14 1B E7 77 C1 30 E7 77 ..C'wC.0C'wb&
	`b& b& b& SECTION[n]`	b&0001BF00 63 00 2E 00 63 00 68 00 c...c.h.0001BF08 6D 00 0A 00 43 00 61 00 m...C.a.0001BF10 6C 00 63 00 75 00 6C 00 l.c.u.l.0001BF18 61 00 74 00 6F 00 72 00 a.t.o.r.0001BF20 11 00 4E 00 6F 00 74 00 ..N.o.t.0001BF28 20 00 45 00 6E 00 6F 00 .E.n.o.0001BF30 75 00 67 00 68 00 20 00 u.g.h. .0001BF38 4D 00 65 00 6D 00 6F 00 M.e.m.o.0001BF40 72 00 79 00 00 00 00 00 r.y.....0001BF48 00 00 00 00 00 00 00 00 ........0001BF50 00 00 00 00 00 00 00 00 ........0001BF58 00 00 00 00 00 00 00 00 ........0001BF60 00 00 00 00 00 00 00 00 ........0001BF68 00 00 00 00 00 00 00 00 ........0001BF70 00 00 00 00 00 00 00 00 ........0001BF78 00 00 00 00 00 00 00 00 ........

2.2 The Windows NT data

As mentioned in the preceding section, e_lfanew storage in the MS-DOS data structure refers to the location of the Windows NT information. Hence, if you assume that the pMem pointer relates the start point of the memory space for a selected portable executable file, you can retrieve the MS-DOS header and also the Windows NT headers by the following lines, which you also can perceive in the PE viewer sample (pelib.cpp, PEStructure::OpenFileName()):

IMAGE_DOS_HEADER        image_dos_header;IMAGE_NT_HEADERS        image_nt_headers;PCHAR pMem;b&memcpy(&image_dos_header, pMem,       sizeof(IMAGE_DOS_HEADER));memcpy(&image_nt_headers,       pMem+image_dos_header.e_lfanew,       sizeof(IMAGE_NT_HEADERS));

IMAGE_NT_HEADERS structure definition. It makes it possible to grasp what the image NT header maintains to execute a code inside the Windows NT OS. Now, you are conversant with the Windows NT structure; it consists of the "PE" Signature, the File Header, and the Optional Header. Do not forget to take a glimpse at their comments in the MSDN Library and in Table 1.

It seems to be very simple, the retrieval of the headers information. I recommend inspecting the MSDN library regarding the

One the whole, I consider merely, in most circumstances, the following cells of the IMAGE_NT_HEADERS structure:

FileHeader->NumberOfSectionsOptionalHeader->AddressOfEntryPointOptionalHeader->ImageBaseOptionalHeader->SectionAlignmentOptionalHeader->FileAlignmentOptionalHeader->SizeOfImageOptionalHeader->DataDirectory[IMAGE_DIRECTORY_ENTRY_IMPORT]              ->VirtualAddressOptionalHeader->DataDirectory[IMAGE_DIRECTORY_ENTRY_IMPORT]              ->Size

You can observe the main purpose of these values clearly, and their role when the internal virtual memory space allocated for an EXE file by the Windows task manager if you pay attention to their explanations in MSDN library, so I am not going to repeat the MSDN annotations here.

I should make a brief comment regarding the PE data directories, or OptionalHeader-> DataDirectory[], because I think there are a few aspects of interest concerning them. When you come to survey the Optional header through the Windows NT information, you will find that there are 16 directories at the end of the Optional Header, where you can find the consecutive directories, including their Relative Virtual Address and Size. I just mention here the notes from <winnt.h> to clarify these information:

// Export Directory#define IMAGE_DIRECTORY_ENTRY_EXPORT          0// Import Directory#define IMAGE_DIRECTORY_ENTRY_IMPORT          1// Resource Directory#define IMAGE_DIRECTORY_ENTRY_RESOURCE        2// Exception Directory#define IMAGE_DIRECTORY_ENTRY_EXCEPTION       3// Security Directory#define IMAGE_DIRECTORY_ENTRY_SECURITY        4// Base Relocation Table#define IMAGE_DIRECTORY_ENTRY_BASERELOC       5// Debug Directory#define IMAGE_DIRECTORY_ENTRY_DEBUG           6// Architecture Specific Data#define IMAGE_DIRECTORY_ENTRY_ARCHITECTURE    7// RVA of GP#define IMAGE_DIRECTORY_ENTRY_GLOBALPTR       8// TLS Directory#define IMAGE_DIRECTORY_ENTRY_TLS             9// Load Configuration Directory#define IMAGE_DIRECTORY_ENTRY_LOAD_CONFIG    10// Bound Import Directory in headers#define IMAGE_DIRECTORY_ENTRY_BOUND_IMPORT   11// Import Address Table#define IMAGE_DIRECTORY_ENTRY_IAT            12// Delay Load Import Descriptors#define IMAGE_DIRECTORY_ENTRY_DELAY_IMPORT   13// COM Runtime descriptor#define IMAGE_DIRECTORY_ENTRY_COM_DESCRIPTOR 14

The last one (15) was reserved for use in the future; I have not yet seen any purpose for it, even in PE64.

For instance, if you want to perceive the relative virtual address (RVA) and the size of the resource data, it is enough to retrieve them by:

DWORD dwRVA  = image_nt_headers.OptionalHeader->   DataDirectory[IMAGE_DIRECTORY_ENTRY_RESOURCE]->VirtualAddress;DWORD dwSize = image_nt_headers.OptionalHeader->   DataDirectory[IMAGE_DIRECTORY_ENTRY_RESOURCE]->Size;

To comprehend more regarding the significance of data directories, I forward you to Section 3.4.3 of the Microsoft Portable Executable and the Common Object File Format Specification document by Microsoft, and furthermore Section 6 of this document, where you discern the various types of sections and their applications. You will see the section’s advantage subsequently.

2.3 The Section Headers and Sections

You currently observe how the portable executable files declare the location and the size of a section on a disk storage file and inside the virtual memory space allocated for the program with IMAGE_NT_HEADERS-> OptionalHeader->SizeOfImage by the Windows task manager, as well the characteristics to demonstrate the type of the section. To better understand the Section header as my previous declaration, I suggest having a brief look at the IMAGE_SECTION_HEADER structure definition in the MSDN library. For an EXE packer developer, VirtualSize, VirtualAddress, SizeOfRawData, PointerToRawData, and Characteristics cells have significant rules. When developing an EXE packer, you should be clever enough to play with them. There are somet hings to note when you modify them; you should take care to align the VirtualSize and VirtualAddress according to OptionalHeader->SectionAlignment, as well as SizeOfRawData and PointerToRawData in line with OptionalHeader->FileAlignment. Otherwise, you will corrupt your target EXE file and it will never run. Regarding Characteristics, I pay attention mostly to establish a section by IMAGE_SCN_MEM_READ | IMAGE_SCN_MEM_WRITE | IMAGE_SCN_CNT_INITIALIZED_DATA, I prefer that my new section has the ability to initialize such data during the running process, such as import table; besides, I need it to be able to modify itself by the loader with my settings in the section characteristics to read- and writeable.

Moreover, you should pay attention to the section names; you can know the purpose of each section by its name. I will just forward you to Section 6 of the Microsoft Portable Executable and the Common Object File Format Specification documents. I believe it represents the totality of sections by their names; this is also included in Table 2.

Table 2: Section names

".text"	Code Section
"CODE"	Code Section of file linked by Borland Delphi or Borland Pascal
".data"	Data Section
"DATA"	Data Section of file linked by Borland Delphi or Borland Pascal
".rdata"	Section for Constant Data
".idata"	Import Table
".edata"	Export Table
".tls"	TLS Table
".reloc"	Relocation Information
".rsrc"	Resource Information

To comprehend the section headers and also the sections, you can run the sample PE viewer. With this PE viewer, you can realize only the application of the section headers in a file image, so to observe the main significance in the Virtual Memory, you should try to load a PE file by a debugger. The next section represents the main idea of using the virtual address and size in the virtual memory by using a debugger. The last note is about IMAGE_NT_HEADERS-> FileHeader->NumberOfSections, that provides a number of sections in a PE file. Do not forget to adjust it whenever you remove or add some sections to a PE file. I am talking about section injection!

3 Debugger, Disassembler and some Useful Tools

In this part, you will become familiar with the necessary and essential equipment to develop your PE tools.

3.1 Debuggers

The first essential prerequisite to become a PE tools developer is to have enough experience with bug tracer tools. Furthermore, you should know most of the assembly instructions. To me, the Intel documents are the best references. You can obtain them from the Intel site for IA-32, and on top of that IA-64; the future belongs to IA-64 CPUs, Windows XP 64-bit, and also PE64!

To trace a PE file, SoftICE by Compuware Corporation, I knew it also as named NuMega when I was at high school, is the best debugger in the world. It implements process tracing by using the kernel mode method debugging without applying Windows debugging application programming interface (API) functions. In addition, I will introduce one perfect debugger in user mode level. It utilizes the Windows debugging API to trace a PE file and also attaches itself to an active process. These API functions have been provided by Microsoft teams, inside the Windows Kernel32 library, to trace a specific process, by using Microsoft tools, or perhaps, to make your own debugger! Some of those API functions inlude:

CreateThread() CreateProcess() OpenProcess() DebugActiveProcess() GetThreadContext() SetThreadContext() ContinueDebugEvent() DebugBreak() ReadProcessMemory() WriteProcessMemory() SuspendThread() ResumeThread()

3.1.1 SoftICE

It was in 1987; Frank Grossman and Jim Moskun decided to establish a company called NuMega Technologies in Nashua, NH, to develop some equipment to trace and test the reliability of Microsoft Windows software programs. Now, it is a part of Compuware Corporation and its product has participated to accelerate the reliability in Windows software, and additionally in Windows driver developments. Currently, everyone knows the Compuware DriverStudio that is used to establish an environment for implementing the elaboration of a kernel driver or a system file by aiding the Windows Driver Development Kit (DDK). It bypasses the involvement of DDK to implement a portable executable file of kernel level for a Windows system software developer. For us, only one instrument of DriverStudio is important, SoftICE; this debugger can be used to trace every portable executable file, a PE file for user mode level or a PE file for kernel mode level.

Figure 1: SoftICE Window

EAX=00000000EBX=7FFDD000 ECX=0007FFB0 EDX=7C90EB94 ESI=FFFFFFFF EDI=7C919738 EBP=0007FFF0 ESP=0007FFC4 EIP=010119E0 o d i s z a p c
CS=0008 DS=0023 SS=0010 ES=0023 FS=0030 GS=0000 SS:0007FFC4=87C816D4F

0023:01013000 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ……………. 0023:01013010 01 00 00 00 20 00 00 00-0A 00 00 00 0A 00 00 00 ……………. 0023:01013020 20 00 00 00 00 00 00 00-53 63 69 43 61 6C 63 00 ……..SciCalc. 0023:01013030 00 00 00 00 00 00 00 00-62 61 63 6B 67 72 6F 75 ……..backgrou 0023:01013040 6E 64 00 00 00 00 00 00-2E 00 00 00 00 00 00 00 nd…………..

0010:0007FFC4 4F 6D 81 7C 38 07 91 7C-FF FF FF FF 00 90 FD 7F Om |8 b.| . 0010:0007FFD4 ED A6 54 80 C8 FF 07 00-E8 B4 F5 81 FF FF FF FF T . 0010:0007FFE4 F3 99 83 7C 58 6D 81 7C-00 00 00 00 00 00 00 00 Xm |…….. 0010:0007FFF4 00 00 00 00 E0 19 01 01-00 00 00 00 00 00 00 00 …. ….

010119E0 PUSH EBP 010119E1 MOV EBP,ESP 010119E3 PUSH -1 010119E5 PUSH 01001570 010119EA PUSH 01011D60 010119EF MOV EAX,DWORD PTR FS:[0] 010119F5 PUSH EAX 010119F6 MOV DWORD PTR FS:[0],ESP 010119FD ADD ESP,-68 01011A00 PUSH EBX 01011A01 PUSH ESI 01011A02 PUSH EDI 01011A03 MOV DWORD PTR SS:[EBP-18],ESP 01011A06 MOV DWORD PTR SS:[EBP-4],0

3.1.2 OllyDbg

It was about four years ago that I first saw this debugger by chance. For me, it was the best choice; I was not wealthy enough to purchase SoftICE, and at that time, SoftICE only had good functions for DOS, Windows 98, and Windows 2000. I found that this debugger supported all kinds of Windows versions. Therefore, I started to learn it very fast, and now it is my favorite debugger for the Windows OS. It is a debugger that can be used to trace all kinds of portable executable files except a Common Language Infrastructure (CLI) file format in user mode level, by using the Windows debugging API. Oleh Yuschuk, the author, is one of worthiest software developers I have seen in my life. He is a Ukrainian who now lives in Germany. I should mention here that his debugger is the best choice for hacker and cracker parties around the world! It is freeware! You can try it from the OllyDbg Homepage.

Figure 2: OllyDbg CPU Window

(

3.1.3 Which parts are important in a debugger interface?

I have introduced two debuggers without talking about how you can employ them, and also which parts you should pay attention to. Regarding using debuggers, I refer you to their instructions in help documents. However, I want to explain briefly the important parts of a debugger; of course, I am talking about low-level debuggers, or in other words, machine-language debuggers of the x86 CPU families.

All of low-level debuggers consist of the following subdivisions:

Registers viewer.

EAX

ECX

EDX

EBX

ESP

EBP

ESI

EDI

EIP

o d t s z a p c

Disassembler or Code viewer.

010119E0 PUSH EBP010119E1 MOV EBP,ESP010119E3 PUSH -1010119E5 PUSH 01001570010119EA PUSH 01011D60010119EF MOV EAX,DWORD PTR FS:[0]010119F5 PUSH EAX010119F6 MOV DWORD PTR FS:[0],ESP010119FD ADD ESP,-6801011A00 PUSH EBX01011A01 PUSH ESI01011A02 PUSH EDI01011A03 MOV DWORD PTR SS:[EBP-18],ESP01011A06 MOV DWORD PTR SS:[EBP-4],0

Memory watcher.

Stack viewer.

Command line, command buttons, or shortcut keys to follow the debugging process.

Command SoftICE OllyDbg

Run F5 F9

Step Into F11 F7

Step Over F10 F8

Set Break Point F8 F2

You can compare Figures 1 and 2 to distinguish the difference between SoftICE and OllyDbg. When you want to trace a PE file, you should mostly consider these five subdivisions. Furthermore, every debugger comprises of some other useful parts; you should discover them by yourself.

3.2 Disassembler

You can consider OllyDbg and SoftICE to be excellent disassemblers, but I also want to introduce another disassembler tool that is famous in the reverse engineering world.

3.2.1 Proview disassembler

Proview or PVDasm is an admirable disassembler by the Reverse-Engineering-Community; it is still under development and bug fixing. You can find its disassmbler source engine and employ it to create your own disassembler.

3.2.2 W32Dasm

W32DASM can disassemble both 16- and 32-bit executable file formats. In addition to its disassembling ability, you can employ it to analyze import, export, and resource data directories data.

3.2.3 IDA Pro

All reverse-engineering experts know that IDA Pro can be used to investigate, not only x86 instructions, but that of various kinds of CPU types like AVR, PIC, and so forth. It can illustrate the assembly source of a portable executable file by using colored graphics and tables, and is very useful for any newbie in this area. Furthermore, it has the capability to trace an executable file inside the user mode level in the same way as OllyDbg.

3.3 Some Useful Tools

A good PE tools developer is conversant with the tools that save his time, so I recommend that you select some appropriate instruments to investigate the base information under a portable executable file.

3.3.1 LordPE

LordPE by y0da is still the first choice to retrieve PE file information with the possibility to modify them.

3.3.2 PEiD

PE iDentifier is valuable to identify the type of compilers, packers, and cryptors of PE files. As of now, it can detect more than 500 different signature types of PE files.

3.3.3 Resource Hacker

Resource Hacker can be employed to modify resource directory information; icon, menu, version info, string table, and so on.

3.3.4 WinHex

WinHex, it is clear what you can do with this tool.

3.3.5 CFF Explorer

Eventually, CFF Explorer by Ntoskrnl is what you want to have as a PE Utility tool in your arsenal; it supports PE32/64, PE rebuild included Common Language Infrastructure (CLI) file. In other words, the .NET file, a resource modifier, and much more facilities which can not be found in others. Just try to discover every unimaginable option by hand.

4 Add a New Section and Change the OEP

You are ready to do the first step of making your project. I have provided a library to add a new section and rebuild the portable executable file. Before starting, I wnat you to get familiar with the headers of a PE file, by using OllyDbg. You should first open a PE file; that pops up a menu, View->Executable file. Again, you get a popup menu: Special->PE header. You will observe a scene similar to Figure 3. Now, come to the Main Menu View->Memory, and try to distinguish the sections inside the Memory map window.

Figure 3

00000000000000020000000400000006000000080000000A0000000C0000000E00000010000000120000001400000016000000180000001A0000001C0000001D0000001E0000001F0000

Computer, Electron and Technology

Search Box

Buzz

Categories

Meta

Blogroll

RSS Feed

Popular Tags

google背后的分布式架构

PHP:set_error_handler ……need more

XCache & XDebug on road

redhad下的openssl(安装和卸载)

OpenSSL相关命令（for Linux）详细介绍

[收藏] Asian NTP Server pool

[超长篇] Inject Your Code to a Portable Executable File

0 Preface

1 Prerequisites

2 Portable Executable File Format

2.1 The MS-DOS data

PE Viewer

2.2 The Windows NT data

2.3 The Section Headers and Sections

3 Debugger, Disassembler and some Useful Tools

3.1 Debuggers

3.1.1 SoftICE

3.1.2 OllyDbg

3.1.3 Which parts are important in a debugger interface?

3.2 Disassembler

3.2.1 Proview disassembler

3.2.2 W32Dasm

3.2.3 IDA Pro

3.3 Some Useful Tools

3.3.1 LordPE

3.3.2 PEiD

3.3.3 Resource Hacker

3.3.4 WinHex

3.3.5 CFF Explorer

4 Add a New Section and Change the OEP

Figure 3

Command	SoftICE	OllyDbg
Run	F5	F9
Step Into	F11	F7
Step Over	F10	F8
Set Break Point	F8	F2