AWS - Running Services Locally (a quick survey of products)

My first step was to review the major services we wanted to use on Amazon. They have a massive set of services available for use and some are not easily replicated without a major infrastructure and skill set to manage it. To give you some idea of the number of services here is a summary of the services and a quick reference as of August 2013.

Amazon Web Services (AWS) as of August 2013 Compute & Networking EC2 Virtual Servers in the Cloud ELB Elastic Load Balancing or Auto Scaling of services EMR Hosted Hadoop Framework VPC Isolated Cloud Resources Route 53 Scalable Domain Name System (DNS)
Direct Connect Dedicated Network Connection to AWS Storage & CDN S3 Scalable Storage in the Cloud
Glacier Low-Cost Archive Storage in the Cloud
EBS EC2 Block Storage Volumes
Import/Export Large Volume Data Transfer Storage Gateway Integrates on-premises IT environments with Cloud storage CloudFront Global Content Delivery Network (CDN) Database RDS Managed Relational Database Service for MySQL, Oracle and SQL Server DynamoDB Fast, Predictable, Highly-scalable NoSQL data store ElastiCache In-Memory Caching Service Redshift Fast, Powerful, Fully Managed, Petabyte-scale Data Warehouse Service
Application Services CloudSearch Managed Search Service SWF Workflow service for coordinating application components SQS Message Queue Service SES Email Sending Service SNS Push Notification Service FPS Amazon Flexible Payments Service (FPS) is an API based payment service Elastic Transcoder Easy-to-use scalable media transcoding Deployment & Management Management Console Web-Based User Interface IAM Identity and Access Management (IAM) configurable access controls
CloudWatch Resource and Application Monitoring Elastic Beanstalk AWS Application Container CloudFormation Templates for AWS Resource Creation Data Pipeline Orchestration Service for Periodic, Data-Driven Workflows OpsWorks DevOps Application Management Services CloudHSM Hardware-based Key Storage for Regulatory Compliance

The items above underscored are those we deemed important or critical to our project.


PaaS (Platform as a Service)

I limited the search to the major players in the AWS private cloud market and ignored the non-AWS compatible options. AWS is the gold-standard of cloud services at this time. Other vendors are catching up to the services they offer.  Google, Yahoo, HP, Microsoft, and EMC/VMware are all attempting to gain traction and provide the services that Amazon currently offers. I don't see this changing in the near future but those options may improve as this market matures. Google in particular seems to want into this space and the EMC/VMWare has the Pivotal initiative with partners. There are other players but Amazon owns this space right now.

With those requirements, OpenStack, CloudStack and Eucalyptus are the three major packages I reviewed. They are listed in order of least AWS services to greatest number of services replicated.  The clear winner was Eucalyptus from reviewing the documentation and frequenting the IRC channels for regular users using the products.

OpenStack has a very robust hypervisor management and API for supporting those services. There is a bolted on AWS API for EC2 that works for the most part until it comes into conflict with the way the OpenStack API functions. Reviewing the usage of OpenStack and participating in the community surrounding it, left me with little patience for this product. It was not a very open community of users and there seemed to be some hostility towards AWS compatibility. My goal is to implement an AWS compatible system so this group does not seem to align with that goal. Likewise their API support is limited to EC2 and a limited subset of other services to support EC2. The OpenStack Compute API is very similar to the AWS EC2 API so many people port between them as necessary. There are even some projects out to assist in this effort that don't appear to be very active now.  An older project by Canonical called AWSOME (any web service over me) was supposed to bridge Amazon and OpenStack cloud environments.

As a side note, the OpenStack versus Eucalyptus debate has the feel of the almost religious debates that pervaded the Debian versus Ubuntu arguments several years ago. The free and open software debate is not my primary concern today but I understand the arguments about having long term freedom. Unfortunately, I need to implement something today and the AWS API is the closest thing to a standard we have for this technology right now. I am not ignorant of the risk that using Amazon controlled standards and Eucalyptus implements copies entails if they close the source but I have faith that a branch will emerge to continue the open source version if that happened like with MySQL and many other software packages over the years. Amazon is working hard to continue to innovate and others are pushing to catch up. So this will stay an active area not letting Amazon slow down. The chest beating between OpenStack and other cloud providers makes them less interesting to me.

CloudStack may gain traction now that it is under the Apache Foundation and separated from Citrix the makers of Xen. I hope they increase the number of services offered but as of today, they are not sufficient for our needs. They offer a limited EC2/S3 service without any of the other services that make AWS so interesting. If you are looking for just a virtual machines management system with an EC2 interface, then this will definitely serve your needs. One major advantage over Eucalyptus is that it offers support for additional hypervisors that the current Eucalyptus does support not including Xen, VMWare, and KVM. This could be the difference for some but was not a factor in our decision. The community was quite open and interested in new comers. If they offered more AWS services, I would have been happy with this product.

Eucalyptus which I began working with off the GitHub repository for the 3.3.0 release candidate contains a large number of complex moving parts. In a prior post on Eucalyptus, I gave a list of features that included the base EC2, S3, EBS, AMI, IAM, and the more recently added Autoscaling, Elastic Load Balancer, and Cloudwatch. This whole package was in flux while I was learning about it and building it so I had some additional self-imposed hurdles. The S3 support is lacking some features but is a decent implementing for storing small amounts of information and a relatively small number of files. It serves fine for storing EMI (Eucalyptus Machine Images) or simple configuration data. The coverage of supported API is decent. Don't do something strange like use the AWS DotNet SDK and you are likely to get it working fine. The EBS support works but requires some extra effort to create the initial images. Those issues are being worked on actively by Eucalyptus and you should see significant changes in the near future.

The EC2 support appears to be solid but is limited to the Linux KVM hypervisor only. There used to be support for Xen which was removed in the last couple of versions. Open source users have gotten it to work with Xen recently but it isn't in the main line support right now.

The S3 support is being bolstered internally in their Walrus service and with third party software like Ceph and Riak CS (S3 compatibility with HA). These are ambitious additions to their existing systems and will likely take a few revisions to work out the issues. You can review their road map to see about when they plan for these features.

There are a few options available for AWS replicated services run locally outside of Eucalyptus. I'll have a follow up post sometime in the next few days on a few of these that will include at least: S3, SQS, SNS, DynamoDB, RDS and SWF. These are services not offered or incomplete implementations on Eucalyptus.

Another post that I will flesh out will be about the shared storage used by Eucalyptus to allow for shared volumes between the various components of the system. Not having a NetApp or EMC storage device available made it necessary to learn a bit about free options in this space.

Please comment or ask questions.

Eucalyptus (AWS private Cloud Computing)

Eucalyptus (AWS private Cloud Computing)

I'm not going to give a full run down of what Eucalyptus is but just point you to their marketing material at their website. The quick summary is it offers the Amazon Web Services loaded on a local computer.  These include several of the most interesting services: EC2, S3, EBS, AMI, IAM, and recently they added Autoscaling, Elastic Load Balancer, and Cloudwatch. If that alphabet soup has your interest piqued, then you should continue reading.

Building one of these using their pre-packaged images is dead simple. I'm not one to do anything the simple way and decided to build everything from the source directly from their Github repository. This was not an easy task but definitely taught me a lot about their software and the components of the system.  I would recommend a first-time user to not take my route and just take their binary builds from RPM or their ISO image. Fedora Core has these as well and the guy who supports it is a great guy. Please take the path of least resistance first to get familiar with the software.

In my configuration, I have a couple five to seven year old servers that used to be production. I've got a system with a twelve (12) core CPU and forty-eight (48) GB of RAM and a second system with dual-core and 4GB of RAM. They are an old database server and an old web-server. The heavy-weight system with the better memory and processor was dedicated to serving out virtual machines and the lower-end system is the web services provider. I had a third system that is just a desktop box that is acting as my SAN device with FreeNAS 8.3.

The front end to the whole thing to isolate it from my network is a cheap wireless router that serves out DHCP reservations and provides a private network. The only smart thing on the router is a firewall with port forwarding. I added another desktop system with Linux installed that I use as a jump host into the environment running OpenSSH.

Added to this is a virtual machine image that is running CentOS 6.4 that runs on the virtual machine server to actually build the Eucalyptus software. This image is running under Linux KVM which will later be used by the Eucalyptus software to serve out images managed by EC2. I subverted the environment to let me use it for a build server as well.

• router - dLink wireless router
• marduk - web services and custom tools
• tiamat - virtual machine provider
• buildserver vm image
• EC2 instances
• anshar - SAN server (iSCSI) running FreeNAS 8.3.1
• gozer - jumphost and utilities server
• OpenSSH
• Nagios service monitoring

This relatively cheap set of hardware components allows me to replicate the Amazon services and test my code locally.

I'm not going to sugar coat it, there were issues along the way.  Here is a list of the ones that pop to mind:

1. I abandoned trying to build software on Ubuntu 12.04.02 LTS and migrated to CentOS 6.4 for the buildserver.
2. The build process is only mostly documented (but much better in 3.3) with some dependencies missing and no separation between build and runtime environment.
3. iSCSI is never fun to configure (but no harder than the regular iSCSI fun)
4. The S3 support is hit and miss (DeleteObjects and multi-part POST API fail) some are fixed in version 3.4 and a major update is coming in 4.0 (Ceph and Riak CS). They are addresssing this actively.
5. iSCSI volumes has strange behavior with KVM virtual machines
1. cache=writethrough necessary for kvm images
2. DAS configuration of Storage Controller takes a couple tries and is a one-way trip
6. I'm still working on windows imaging (painful but getting better) and the 3.4 will have significant improvement in this area. Eustore may be on option soon.
7. They are still working on bfEBS (bootable EBS) but check the IRC channel for help. It works with caveats.

On the plus side their support is excellent and I would recommend joining their IRC channel. Also, some of the above issues had to do with my learning curve. I could have used the Fedora provided RPM or their FastStart image and have gotten much further quicker but I'm stubborn when I start working on something.

I hope someone can use this later.  I'm trying to write down the entire process of building this system and will post it back out here later when it looks a little better. I've got a lot of documentation that needs to be cleaned up for release so the learning curve is reduced.

On a completely separate note, I've got a quick-and-dirty Grails application to allow for viewing and managing Eucalyptus S3 components and it's pretty cool to have a local repository to play around with before paying for the AWS service. I may post on that later when I get time to clean it up a bit. The code is a mess as I was hacking it together to help diagnose issues with S3 and write clean code.

Amazon Web Services

New job and learning AWS (http://aws.amazon.com) in support of distributed computing. I've done a good bit of research to understand the ecology of web services. They can be expensive but when you count total costs, they come out pretty close to break-even. You can lease computational power for relatively short periods to do quick work, then give the hardware back and stop paying for it. That is a powerful shift in technology.

OpenStack, CloudStack, Eucalyptus and a couple others where in the running while doing the evaluations. Each had their advantages which I'll probably write about later.

I settled on implementing a Eucalyptus system (http://www.eucalyptus.com) using spare hardware. So far I've got a working and running system built from the source code. It provides a subset of the AWS services on local hardware for testing.  The S3 (http://aws.amazon.com/s3/) support isn't quite there as of version 3.3.0 but it might improve in the next couple of months. S3 is simple storage interface allowing for storing information by a key value.  Their EC2 (http://aws.amazon.com/ec2/) support appears to be much better and allows for quickly building virtual machines with pre-configured operating systems.

In the background, I'm writing Java code to implement AWS tools and services. I forgot how much fun Java can be.

DLNA with Sony

Bought a Sony Blu-ray player with DLNA support. Added MediaTomb with a static binary on the Blackarmor. Tested it an it wasn't very happy. It did run but the performance just wasn't there.

The default Sony Blu-ray with my new BA NAS 400 series works just fine. A solution to my long standing problem with playing my digital content on my TV and sound system. DLNA support in the Sony equipment just seems to work.

Cygwin Ports by way of binary files

Binary Files

I wanted to do some work on an undocumented binary file format. The free HexEdit tool for Windows from Catch22 Software is a pretty good for mapping data structures to the raw binary/hex data in a file.  It uses the C format struct to format the data structures into a human readable format.  It could use some additional docs on how to use the typedef format but was relatively easy to figure out.

What was frustrating was finding a tool that would allow for doing binary diffs with a decent interface.  There are several methods for doing binary diffs on files.  The unix 'od' command, 'bvi' and others presented themselves but were not as interactive as I would like when looking at large numbers of files visually for iterative small changes. Editors and Binary Editors fall into an area of preferences. I could be entering a flame-war picking a tool but would love to hear some feedback on tools people have used.

What I found that seemed to fit the bill for me was Chris' excellent tool called vBinDiff found at his website. Since I wanted to use UNIX tools for scripted automation but still be on a Windows system, I installed Cygwin.  Getting vbindiff to work on Cygwin was a testament to Chris' excellent code base.  After finishing up and getting it working, I thought it might be nice to package it for the Cygwin project.

Cygports

Thus started my journey into Cygport.  I found myself frustrated with the documentation for the assumption that I knew a lot about the Cygwin packaging which is what I was trying to avoid by using the tool. This is not a bashing session for cygport which does an amazing job of wrapping up lots of the trivia that is Cygwin packaging.  What was missing is a basic "HowTo Package ABC" for Cygwin.  The cygport documents while good are written for someone who had done the manual process of packaging at some point.  The documentation failed to meet my expectations as a competent developer who has done lots of porting work to quickly introduce me to this new means of packaging for their platform.  There is not quick guide on the simple cases with a couple of interesting example.  The other lack was how to get these example applications of varied type out of Cygwin easily.  I figured it out and afterwards felt silly for not having seen it.  All that said, I have come the realization that I have to put some time aside and try to write an HowTo guide on packaging for Cygwin.

I've got two packages done so far.  vBinDiff is the first and I am still polishing that install as I learn more along the way. vbindiff is currently working great as a package on my local system. Wy60 is another tool from years back when I worked on UniData/Universe systems and needed access to their Wyse 60 PICK interfaces. This makes for two packages and I have used several features to make these work.  I'll have to learn how to submit a request for inclusion into the main Cygwin package group.  There is a mailing list and a set of questions to answer.  I'll just need to put some time aside for it. :)

Interests 

So those are my current interests in the world of technology.  I keep playing around with the BlackArmor NAS devices occasionally and would like to find the time to get the USB to serial interface setup on my second unit. The ARM gcc tool-chain languishes for lack of time as well. The DLNA server ended up being too much of a pain without a decent tool-chain so I knuckled under and installed Microsoft Media Center on an old laptop to feed my digital media to my xbox360. I really want to revisit that particular issue and get a low-power digital media server working.  Maybe I'll find the time and pick those back up.

Debian on BA NAS 110

Hajo on the BlackArmor Forums has an older posting about getting Debian Linux 5.0 (Lenny) installed on BA NAS 110/220/4x0. This is not a port that includes the kernel but simply a minimum install that gets the system setup to install binaries out of the Lenny EABI ARM platform.  The kernel that comes with the BA NAS is compatible with those binaries.  The newer kernel for the Debian 6 or higher is not compatible with the BA NAS. This has some limitations but offers a way to get to some newer software pre-compiled.  I don't want to loose the existing functionality on my test system but the draw to DLNA services is pretty strong right now.

To top it off, Debian has a nicely setup cross-compilation setup documented for people working on non-Intel platforms.  This offers a way to compile newer software without killing myself anymore on building the entire compiler and supporting software myself.

The goal has always been to make the NAS device useful and I want to play my movies off it to my TV upstairs so this might be the next thing I play with on the development NAS.

miniDLNA

I was reading a bit more on DLNA servers and found that someone had gotten miniDLNA working under the Debian port for the BlackArmor NAS.  NickolasZev looks like he is enjoying his Debian install on his BA110. He is documenting his Debian initial install, adding miniDLNA, a webserver, bittorrent and download manager. I'm not interested in putting Debian on mine and want to stay as close to the vanilla firmware as possible but it is nice to see that someone has it working on the hardware.

DLNA server for BlackArmor NAS

Today I was reading a couple of forums and bumped into a write up on the "BubbleUPnP - UPnP/DLNA Control Point and Renderer" application for Android that has both DLNA client and server support. The BubbleUPnP setup also has a Java based server mentioned in their section on NAS devices, such as the QNAP which is very similar to the BlackArmor NAS.

Earlier in my writing, I was reviewing a version of Java for the ARM that was called "Sun Java SE for Embedded".  It would allow for executing a Java application on the BlackArmor.  Up until now, I did not have a Java application to really motivate me to install and test this but the BubbleUPnP DLNA server might allow for my BlueRay Play, Roku and Android cell phone to play my movies directly off the BlackArmor NAS.

There is one problem which is getting the ffmpeg library compiled and running. This is described in the docs above for the QNAP NAS but I've not been successful at getting a working toolchain yet.  I may have to backtrack and see if I can just use the toolchain provided by Seagate to compile this newer version of ffmpeg.

Another issue is that the Java SE for Embedded is licensed strangely by Oracle. It cannot be redistributed and requires licensing costs if it is redistributed by a professional package.

So step one would be to compile the ffmpeg library and test it out.  I hope to try that out with the existing Seagate toolchain.  The next step would be to get the Java SE Embedded to run a basic HelloWorld application on the console.  After that, try to get the BubbleUPnP server running.

I'll see if I can get a couple hours together to work on this this coming week. It would be nicer to have an open source DLNA server but I'll take what I can get.  Since I'm talking about taking some time to compile C/C++ code anyway to get this working, I checked on open source DLNA servers and popped up something from eLinux that was interesting to read over.  It looks like MediaTomb, uShare, and Serviio are candidates that could replace the BubbleUPnP server if I get the compiler toolchain working.

We will just have to see how much time I can get together to play around with this.

Junking Hardware / Cleaning Office

I'm clearing out about 90% of my old hardware from my home office.

Things like my ancient Sun Sparc Station 4 and 5 for my old Oracle work, and the older Sparc Classic (lunchbox), Sparc IPX systems used in my distributed web development days, and the Cobalt RaQ and iMac G3 systems I did development on about a decade ago.  I'm also dumping a Compaq 286 laptop that served me well in doing assembly and C coding and it's cousin the 486 that was my constant companion for several years. The dumping continues with numerous boxes of old cabling, strange and exotic old cards (ISA, PCI, MCA, EISA, VESA, Sbus, and some I don't even recognize now), a couple of engineering sample video cards, some really old KVMs, couple of 24 port serial port concentrators (that I hacked the firmware on) and even a pair of old Cisco 2500's used in my network testing days. It's about 20 years of hardware exiting the house before we do the renovations and addition to my home.

It is really sad to see some of these little guys leave the house but looking forward to a cleaned out home office that I can get back to being a pleasant work and coding environment.  So out the door they go.

In dumping all this equipment, I've been gathering the hardware junk box that I can re-use in other hardware projects.  So far I have two of these boxes with weird stuff like an old Palm 5 docking station, some ear phones, some older coaxial cable, and the other random assortment of hardware junk you find useful when hacking hardware. These two boxes are starting too look really strange like something from a Mad Max movie.

This is cathartic writing about all these old pieces of equipment that were used in my professional and personal learning processes. It's hard to part with some of these because of the happy memories associated with figuring out the next *big* problem in technology for that time.

Good bye little friends and now to start gathering the next set. :)

UPS and Print Server

I  bought a APC Back-UPS ES 550 awhile back from CostCo without knowing for sure if it would work with the Seagate BlackArmor NAS. It works great so far. I was afraid I would have to hack on the configuration file for the software but it worked out of the box without any extra effort other than enabling the UPS Manager.

I had a second dilemma which was that I use the NAS as my print server for an older Brother MFC 7420 printer and did not want to loose that configuration.  There is only one USB port off the back of the unit that is supposed to work with printers and UPS.  So for my next surprise, I found that it works for both the UPS and printer with an old un-powered notebook USB hub that I got as a door prize at some Cisco event.

I may test plugging up my old 500GB Seagate USB hard disk to that hub as well and see if it recognizes the space and makes it available.

So for anyone else, I cannot vouch for all UPS and printers but my configuration above is working great.

Serial Port on BlackArmor NAS

I found my old Samsung x426 USB cell phone programmer cable as I was cleaning out my home office. I might have mentioned this cable earlier when I wrote about adding a serial console to the BlackArmor NAS.  The cable is a really old style one that has a weird connector for the cell phone way before the mini-USB became standard. The interesting thing about this cable is that it has a serial to USB converter chip (2303HXC 0546) that does the magic of converting serial to USB. That is why that thing was ridiculously expensive ($35) when I bought it back in the day.

Why this is even on my radar as possible is that the website CrapNAS had an entry for how to connect a serial or USB cable so you can watch the Linux boot up as a serial console session. They specify two different ways to do this. For the serial connection, they specify a MAX3232 as necessary.  The USB connection has a schematic that includes a 2303HX. I'm not sure what the difference is between that and my 2303HXC so I will be doing some reading before I cut into the cable and get out the soldering iron.

Why am I even messing around with a serial console?  Because as I am planning to mess around with the lower level system, I should have a back out plan if I do something wrong. A serial console on the device gives me more options during the boot up even if I cannot connect via a network connection.

As far as getting time to work on the compiler toolchain, I've been relaxing over the holiday break with family, cleaning my messy home office and have not even booted up the virtual machine since my last post. I intend to get back to it someday.

Seagate Black Armor NAS

Yeah it's been hectic at work and lots to do in domestic life as well so this hobby project hasn't gotten much attention. Christmas break is coming however and I'm planning on getting some time to work on it. I think most Open Source projects or hobbies get a boost over the holidays.

Merry Xmas & Happy New Year in case I don't get back here before the holidays.

Newlib error during compile

Earlier I was working on the problem with compiling using the arm-elf-gcc on very basic programs.  After some work, I found the base problem is that the libc replacement from newlib are not being picked up at compile time and properly using the syscalls mechanism. Hardwired hacks to work around it produced an executable but something is still wrong with how the compiler was built.

I've distilled the error down to a single search.

Google Search: "/lib/libc.a(lib_a-exit.o): In function `exit':" "newlib/libc/stdlib/exit.c:65: undefined reference to `_exit'"

Several other people appear to have run into the same problem, so I'll be reading up on the problem and see what solutions exist.

Compilation failures

The problem is that I can get an ARM executable but not sure why the default system is not working to produce a basic HelloWorld.

$ cat test.c
int main (){return 0;}

$ arm-elf-gcc test.c -o test

...
/home/mcgarrah/DevelToolbin/binaries/arm-4.4.6/bin/../lib/gcc/arm-elf/4.4.6/../../../../arm-elf/lib/libc.a(lib_a-exit.o): In function `exit':
exit.c:(.text+0x54): undefined reference to `_exit'
collect2: ld returned 1 exit status

$ arm-elf-gcc test.c -o test -v

This spews a couple of pages of additional output which points me to something called the "Using built-in specs" and lots of directory path information for various things like include files and libraries. These all look about right. A directory /DevelToolbin/binaries/arm-4.4.6/arm-elf/lib has some interesting lib files in it.

$ arm-elf-gcc test.c -o test ~/DevelToolbin/binaries/arm-4.4.6/arm-elf/lib/redboot-syscalls.o

$ file test

test: ELF 32-bit LSB executable, ARM, version 1, statically linked, not stripped

When you are facing a missing library like the above issue, you typically just have to find the right library and add it to your build.  In this case we are missing something so fundamental that we need to figure out why it is broken or will hit future issues.

While finding the above syscall libraries, I noticed some files called specs files which are:

• linux.specs
• rdimon.specs
• rdpmon.specs
• redboot.specs

RDP, RDI, RedBoot and Linux are all syscall (system call) protocols. A syscall protocol describes how a libc (standard C library) communicates with the operating system kernel. For our case this library is newlib which uses another syscall protocol called libgloss as an interface between libc and the above syscall protocols.  I'm not sure which protocols are the default but something is not right about this combination.

$ arm-elf-gcc -dumpspecs

This dumps even more output that is even more cryptic but looks important when taken with the above information. There are built-in defaults for GCC that are being overridden by these specs files.

There is an options to change the specs entries from the GCC command line which I use to identify what is happening.

$ arm-elf-gcc test.c -o test -specs=redboot.specs

$ arm-elf-gcc test.c -o test -specs=pid.specs

$ arm-elf-gcc test.c -o test -specs=linux.specs

The pids and redboot files are a very minor difference in a setting so are essentially the same.  Linux is significantly different as a system call interface.

$ arm-elf-gcc test.c -o test -specs=rdimon.specs

$ arm-elf-gcc test.c -o test -specs=rdpmon.specs

Both the RDI and RDP return basically the same error as the above missing library.  So we have identified what is probably the default libraries used by libgloss.

This is just a journey down the rabbit hole. So I need to revisit the newlib build process and see what I did wrong in it.

Update December 4th, 2011 7:30pm: What you see above happens in both the gcc 4.4.6 and gcc 4.3.2 compiled software sets. Same exact problem for both versions of gcc and associated other libraries. I did a full rebuild of the 4.3.2 and did some tracing of the path built into gcc and re-verified paths to all libraries. Same issue still happens. So we have a genuine problem in the build process that is impacting the compiler.  Could be in any place across the binutils, newlib or gcc compilations. So I'll be digging into each. The nice part is it looks like a common problem so if I fix it in the 4.3.2 series then it will probably fix in the 4.4.6 series as well.

Toolchains compiled

Two full toolchains built and a third that I still think might be made to work.  The first is using older versions of everything and was mostly done as a test to get the build environment working against known sources that are known to build. Even this known build process required some effort to get working in a current OS environment. Those docs, notes and scripts will be coming in the near future.

So to outline what works and not, I give you the following.

Toolchain that comes from older versions of software and the docs from Tom Walsh:

• binutils-2.19.1a.tar.bz2
• gcc-4.3.2.tar.bz2 (with a patch from Tom)
• newlib-1.16.0.tar.gz (with a patch from Tom)
• insight-weekly-CVS-7.0.50-20091130.tar.bz2

Newest versions that compiled based on Tom's scripts:

• binutils-2.22.tar.bz2
• gcc-4.4.6.tar.bz2
• newlib-1.19.0.tar.gz
• insight-CVS-20111130.tar.bz2 (pulled from CVS head and required patching by me)

Newest versions that fails to compile in GCC in zlib:

• binutils-2.22.tar.bz2
• gcc-4.6.2.tar.bz2
• newlib-1.19.0.tar.gz
• insight-CVS-20111130.tar.bz2 (pulled from CVS head and required patching by me)

The issue in GCC is well documented (if you know what you are looking for) as a bug in the "--enabled-multilib" during the build. The zlib library that is packaged with the GCC source fails to build in a cross-compiled configuration. Who knew that GCC packages their own copy of zlib in the GCC sources? There appear to be a couple of solutions which might fix the problem. The first is to just use the native zlib from the host system and pass in "--with-system-zlib" but that feels like a hack instead of a fix. The other is to revert a change in GCC that is documented in a couple of places (Bug45174 and Bug43328). This is a bug in the "configure" phase of the standard "configure;make;make install" but shows up in the "make" stage. So, I'll revisit this as time permits and see about getting the latest GCC 4.6 series working.

The more exciting thing is that it looks like both GCC versions that compiled will compile code to an intermediate state.  That is not proof that it generates a working executable but it is a step in the right direction.

For the GCC 4.3.2 version here is a test showing it compiling a quick test.

$ cat > test.c

int main (){return 0;}

Ctrl-D

$ ./arm-elf-gcc -Os -S test.c

$ cat test.s

        .file   "test.c"

        .text

        .align  2

        .global main

        .type   main, %function

main:

        @ args = 0, pretend = 0, frame = 0

        @ frame_needed = 0, uses_anonymous_args = 0

        @ link register save eliminated.

        mov     r0, #0

        bx      lr

        .size   main, .-main

        .ident  "GCC: (GNU) 4.3.2"

For the GCC 4.4.6 version here is a test showing it compiling a quick test.

$ cat > test.c

int main (){return 0;}

Ctrl-D

$ ./arm-elf-gcc -Os -S test.c

$ cat test.s

        .file   "test.c"

        .text

        .align  2

        .global main

        .type   main, %function

main:

        @ args = 0, pretend = 0, frame = 0

        @ frame_needed = 0, uses_anonymous_args = 0

        @ link register save eliminated.

        mov     r0, #0

        bx      lr

        .size   main, .-main

        .ident  "GCC: (GNU) 4.4.6"

While this is good news, it is not an ARM executable. My followup post will not be so upbeat.

Cross-compiler toolchain update

I've been working on building a toolchain using the notes from OpenHardware Building the ARM GNU 4.3.2 with some success. I finally got the base set of GCC 4.3.2 tools to build successfully. I have not used the resulting GCC to produce an ARM executable or verified the executable works on the Black Armor NAS. Those are tests for tomorrow evening when I can get the NAS setup again on the network. It is currently in a box in the corner.

There were several minor things that needed to be updated and modified to get the scripts and environment to work. I've kept careful notes and will post those in the next couple of days once I've tested the output from the compiler works. I'm also attempting to update the versions of the libraries and software to more current versions as well. The GCC 4.3.2 and associated libraries are several years old and I'm trying to get the GCC 4.6.x to build along with newer newlib, binutils and insight/gdb using the same basic set of notes and scripts. I bumped into a zlib issues in the second phase GCC build that stumped me for the night. I'll hit it again tomorrow. Again, I'm keeping careful notes and build docs for the newer version as well.

The dependencies from the operating system are sometimes a pain to track down for the software. I picked a very stripped down install of Ubuntu. The operating system I am using is Ubuntu Server 11.10 because it is easy to install and update. Any Linux would do but the package names may change. Ubuntu Server has no frills so you add everything you need which means all the libraries like GPM, etc.

So there is some progress and in the next couple of days I'll let you know if the build produces working ARM executables. I'm really excited about getting a working "HelloWorld" out there.

Making Sea Salt

I did something completely different. I didn't boot my laptop the entire Thanksgiving weekend.

I went to the beach, ate steamed oysters, grabbed several gallons of water, spent time with my wife, filtered the water, watched television, and boiled the water to get sea salt. I now have a nice little container of sea salt flakes from my favorite beach and a salt slurry.

I read a couple of web pages on doing this and most of what they say is common sense. Filter the water to clean out the sand and other impurities. Boiling is recommended to kill off any nasty organisms. Getting the water from a beach that is not polluted is recommended.  Also do not collect just after a rain. The resulting salt is a mixture of salt flakes similar to kosher salt and crystals like the fancy salt mills use. I may take the salt slurry and add a bit of water and put in a shallow pan to crystallize so it is nicer looking.

I'm considering making some of this into Christmas presents to family who like our little beach cottage.

Now that was something completely different.

Building the GNU ARM Toolchain: Part 2

Lesson learned on doing a toolchain build or anything else for that matter; make sure you are reading the most current documentation available.  I was working with very old versions of the software by using the GNU ARM website mentioned earlier.  In digging into the problems for those builds, I found a few other sites that have detailed discussions on building the toolchain for specific version of the ARM platform.

One of interest is the OpenHardware Building the ARM GNU 4.3.2 that has lots of useful hints.  Also, the Installing Gnuarm ARM Toolchain on Ubuntu 9.04 had some useful notes as well.  Both are very different approaches and for very different ARM platforms but they have lots of useful notes on how to create a toolchain.  I'll probably use something in the middle between the two to get the job done.

So I lost some time with old versions but learned a good bit about the software in the process.

The tools for the toolchain are:
binutil: low level tools for manipulating object code such as the assembler and linker tools
gcc: GNU compilers which has C/C++ and other tools related to compilation of code to object code
gdb & insight: debugger allowing for finding problems in code
newlib: a standard C library for embedded systems

Each tool builds forward until you have a complete toolchain that allows for creating programs for the platform. We are doing something even more interesting called cross-platform compilations. Since I don't have an ARM platform that I want to build a C compiler on and all the other tools, I am building those tools on my Intel Laptop under Ubuntu 10 LTS.  This means my Intel x86_32 processor will be running a compiler that will output an ARM executable. So the compiler is built to run on x86 but produce ARM.  This is cross platform compilation and is very typical for embedded systems work.

The problem I'm encountering is that I don't want to just use an existing binary build of the toolchain but produce one myself.  The age of the toolchain provided by Seagate is such that even if I wanted to find those versions of the software, I'm unlikely to find them and get them working.  So I'm looking for build a newer version of the compiler and see if the software will run afterwards.

My first program produced will likely be just a simple HelloWord app but from small things come larger ones.  Building OpenSSH, rsync and other tools will follow quickly if I get the toolchain up and running.

Building the GNU ARM Toolchain: Part 1

I found the GNU ARM Toolchain website awhile back and that they have several different versions out there for the toolchain. A toolchain is just the basic tools needed to build software.  In this case it is the standard libraries, the compiler, debugger and various other tools needed to write software.  The version of the toolchain that is provided by Seagate is version 3.0 and a very old version.  The oldest on the ARM website is 3.3 with 4.1 being the newest.

I've pulled down the 3.4, 4.0, and 4.1.  The 4.1 does not compile on my which is Ubuntu 10 LTS and may have to do with x86 versus x86_64 differences.  I'm not sure enough to diagnose yet so I dropped back a version to 4.0 and will check again.  The error in the 4.1 build was in the assembly code opcode section so that isn't an area I want to try debugging at this point.  The version 4.0 of the toolchain software is building cleanly for the initial libraries then failed in the GCC compiler. Sometime it pays to drop back a version to avoid bleeding edge pain and sometime you just find new pain.

Status:
So for 4.1: we have opcode errors in binutil which is very early in the build process and the errors look nasty.

For 4.0, we have an error in GCC "fcntl2.h:51: error: call to â__open_missing_modeâ declared with attribute error: open with O_CREAT" which looks like an error encountered before from some quick google searches. Not spent enough time yet to consider this a loss.

Building the toolchain makes it easier to understand the toolchain provided by Seagate even if it is an older version. Once I get done, I'll write up some details on a the process when I get it working right.

We'll see what happens with building the toolchains over the next day or so.

Sun Java SE for Embedded Systems (Jazelle DBX)

Earlier I mentioned a technology called "Jazelle DBX" for the ARM processor that allows for Java Bytecode eXecution (JBX) directly in the ARM hardware which should make it run faster. That DBX technology is being phased out with newer Thumb-2 instruction set being the new preference by ARM for acceleration.  However, the processor in the BlackArmor NAS was the first processor to have this Jazelle DBX feature and I want to see if it has any merit. I did some digging around, like I mentioned I would, and found that Sun had produced a version of Java that may have this technology.

Sun has two small version of Java.  Once is called Java ME (micro-edition) and the other is Java SE for Embedded. The micro-edition was only for really tightly constrained environments like older cell phones that only had 8MB to 16MB of RAM. It was a feature reduced subset of Java with lots of limitations to make it work in that environment. Remember my old Motorola Razor cellphone from earlier posts and that is where this version of Java lived. That version of Java was crippled and never really seemed to take off.  On the other hand that version of Java is in some of our BlueRay players so it wasn't all bad. We just don't want this version of Java on the BlackArmor as it doesn't give us any interesting things other than cell phone Tetris.

The other version of Java is called "Sun Java SE for Embedded" and lifts many of the limitations of the ME version of Java.  It is a mostly full implementation of Java and allows for most libraries to be used from the reading. The downside is that it requires licensing when being used by a business. Fortunately, development work is free. I pulled a copy of the software as a tarball from Sun and will be taking a look at it when I get an environments setup and some times to play.  It has some requirements that may make is hard to use as it takes a minimum of 32MB of RAM per virtual-machine. Remember that the BlackArmor, only has 128MB total of RAM so that is a quite a lot of memory for just one Java VM.

We'll have to see if this is even feasible but it may open up a huge number of possibilities when you look at the diversity of Java code running out there.

Well that was my fun reading for the evening.  I hope you enjoyed my brain dump or at least found it tolerable. Anyone with experience in the area, please drop comment.