Mergesort in Erlang

Looking around for a language that would make sense for my next project, I’ve decided to look into Erlang.

  • It’s a functional programming language.
  • Predated Scala and the new wave of functional languages. In fact those languages are probably more influenced by Erlang.
  • Inherently designed to support concurrent distributed processing on a large scale, although I doubt it can do the fine grained multicore shared memory stuff Julia could do.
  • More suited to tasks that require parallel processing using message duplication as the primary way to share pass information. Don’t use it for computationally intensive stuff.

So in the tradition of not many other first programs in a new language I wrote a merge sort routine

Some of the funky stuff I bumped up against:

  • Seemingly bizzare usage of ,, ; and . as statement terminators. It reminded me of programming in OCaml.
  • if statements aren’t like anything I’ve ever seen before.
  • Nice pattern matching. Love pattern matching in a language; it lets you break apart the shape of the data structure so easily.
  • Concurrent process handling is so sweet.

nvidia driver bugs and segfaults

I’ve bought some new hardware and installed Arch then saw a lot of this lately:

Coredump: /var/lib/systemd/coredump/core.vim.0.7d23765674024e44955c742e3543820a.764.1452531253000000.lz4
Message: Process 764 (vim) of user 0 dumped core.

Stack trace of thread 764:
#0 0x00007fc5397e4937 kill (
#1 0x0000000000530d90 mch_exit (vim)
#2 0x00007fc5397e4680 __restore_rt (
#3 0x00007fc539b677e0 __lll_unlock_elision (
#4 0x00007fc530cf3ccc n/a (
#5 0x00007fc530c81252 n/a (

Looks like there’s a solution here:

I’ll have to download the beta driver and check.

Xnodes – Architecture of a Scalable Distributed System

Note that this document is a work in progress and represent my ideas about how a scalable distributed HA system should work.

What is Xnodes

A distributed highly available heterogeneous manageable system.

  • Distributed. Nodes in the system can be geographically distant. They need not be on the same local area network.
  • Highly Available. Services running in the system are highly available. The loss of one instance of a service does not affect the delivery of the service. Overloading one instance of a service does not block clients from using that service.
  • Heterogeneous. Nodes in the system do not have to be on the same hardware or same version of software.
  • Manageable. Facilities provided for the installation of applications, their upgrade and downgrade with preservation of transforms to that applications configuration.

Availability or 99.9999% uptime
Say we want to have a highly available service. Let’s pretend it’s a database. We want it to tolerate a failure on the host machine so that it is still available. Simplest solution is to have it running on another host machine that acts as a standby. But what if numerous requests come in thus overloading the active database? Well let’s make the standby node active as well. Now we just need a load balancer at the front that round robins connections from clients between the two active databases. Simple. No need for overly complicated redundancy policies. Now what if we want to scale out to three node in our cluster? Easy we just add another node, make it active and configure our load balancer to round robin connections between three nodes in our cluster. If one node dies the other two will pick up the slack. Again no need for complicated redundancy groups. Availabilty is ensured by having multiple instances of the service running at any one time. If the services need some sort of logic to decide who is the master instance they can build themselves or Xnodes can provide a service that makes that decision.

Data consistency
Ok great so we have a cluster of nodes and we have a way to ensure that a service is always available. But what about the configuration data of that service? In other words, which node is the authority about the information stored? And if we have a service running on multiple nodes surely all instances of the same service should share the same view of the data? Surely we need consistency between values between databases? I say rethink that assumption. How do you think Facebook and Amazon ensure such high availability and scalability? (Hint: they don’t ensure consistency, only that eventually the system will tend to a consistent state) It’s up to the application designer to decide which part of their data they definitely want to keep consistent and in the case where there is contention, it’s up to the application to handle it. In short it becomes a user interface problem. This model is not as bad as it sounds and in practice is far better than trying to enforce consistency on a database that has to scale and be available. The alternative, forcing consistency, requires every client application to agree to the transaction that is about to process, thus slowing the responsiveness of every application since they all have to wait. This has the side effect of reducing availability. Even if the client has established a connection with the database service but it’s waiting for the database to answer another clients request, that database is effectively offline. This has been thought through in the past; if it hadn’t Amazon, Google, Facebook etc would not be in business. Even git uses this conceptual model. Every time we use git ( we are sacrificing consistency and gaining availability and scalability. We sacrifice consistency because we can make changes to out local repository without informing the upstream repository. When we commit a change locally, the local and upstream repositories diverge. When someone else commits a change locally, all other local repositories are no longer in sync. Importantly git doesn’t take any steps to enforce consistency. Consistency happens as part of the workflow and as part of the user interface. The users of git bear the responsibilty and do the work required to maintain consistency. We know that our local and upstream repositories will eventually be consistent because after everyone pushes their local commits, and everyone pulls from the upstream repository, all repositories will be consisitent. Similarly Apache Cassandra is a database that does what I’ve just outlined: All nodes will eventually tend towards a consistent state. The has this to say: “Cassandra weak consistency comes in the form of eventual consistency which means the database eventually reaches a consistent state. As the data is replicated, the latest version of something is sitting on some node in the cluster, but older versions are still out there on other nodes, but eventually all nodes will see the latest version.” In Cassandra’s case the applications, and not the user as is the case with git, bear the reposibility of resolving the conflicting case.

Hypothetical Use Cases

To give you an idea of what Xnodes helps with the following use cases are outlined below as a thought experiment.

Running a web server
The typical scenario is an organization running a web server that delivers content to external clients. Xnodes will manage changes to the configuration of the server. Transforms to the services config are preserved so that a rollback can be performed. Additionally a snapshot of the content associated with that instance of the config is taken. This allows a proper rollback of the data. It’s not sufficient to only record the configuration changes. Dependencies between configuration and associated binaries are also tracked. For example assume that the Apache Web Server (httpd) has a dependency on OpenSSL. The current system is running version A of httpd and version A of libssh2. Then one day OpenSSL implements version 320 of the TLS protocol and release version B of openssl. Later, the Apache developers release a version, B, of httpd that supports version B of openssl. They also add another key in the httpd.conf configuration file. This means that this config can only work with version B of httpd and requires the new version of the openssl shared object on the host machine. Anyway the system admin performs the upgrade, installing version B of httpd and version B of openssl and updating the httpd.conf file to use the new configuration option. A few days later, for some reason, they decided to rollback httpd. Maybe and exploit was discovered in the latest version of httpd, something they couldn’t test for before upgrading. Xnodes allows them to do this safely because the configuration data associated with version A of httpd is preserved. This state becomes the state that all nodes in the system will tend to. Content can also be snapshot to the version of the software installed.

Installing software
Installing a new piece of software changes the configured state of the system. Software is first installed on any node in the system and then the other nodes follow that node automatically install the software. In cases where the installation fails an alarm is raised.

Upgrading changes the configured state of an existing software package. The old state is snapshot (including config data and content), the binaries updated and new config replaced. Configuration transforms are executed. Again this is done on a single node in the xnode system first. All other nodes then tend to this nodes configuration.

The software binaries are downgraded to the previous version. The previous configured state is restored. Other nodes tend to this configuration.

The application binaries are removed from the running instance of the node. Configuration state is preserved. All nodes will tend to this new state. A system admin can always rollback the system config to a time where the application was installed.

In production testing
Say you want to role out the latest version of WordPress. You have an Xnode system running but want to test this latest version of WordPress. Xnodes will allow you to 1) select the nodes to isolate 2) role out the upgrade to WordPress on isolated nodes. Then you can test the new version of WordPress by configuring your network to redirect a percentage of requests to the isolated set of nodes.

Configuration data

The Xnodes in memory database is simply a key value database. It is intended to store system configuration information, like the packages installed on the Xnodes system, their configuration. Changes to the configuration are not immediately synchronized. By trading consistency this allows Xnodes to provide scalabilty and availabilty. The following is a quote from the OpenSAF IMM Programmers Guide and describes a design Xnodes DOES NOT implement:

“During the time when sync is in progress, the IMMSv is not writable. The IMMNDs will reject any request that implies a mutation to the repository with SA_AIS_ERR_TRY_AGAIN. Read requests are of course still serviced. Updates to cached non-persistent runtime attributes are also allowed during sync.”

Open Issues

How are services that require redundancy but every instance of the service can’t be in use handled? For example how would Xnodes handle a dhcp service?

Septic Arthritis

Septic Arthritis (these events happened in March 2007)

I woke up shivering. I was in London at my cousin Selvan’s place, lying in bed wondering why I felt so awful. I moved, tried to get up, but my left leg hurt. It would not bend properly. The shivers passed through me in a constant wave making my teeth snap. I limped over to the shower careful not to put pressure on the left leg, careful not to bend it too much. It was hot from the calf to the knee, visibly swollen and red. Turning on the water and standing in the shower, balancing on my good right leg, provided temporary relief. I could breathe again.

I spent 3 days trying to get over the fever. The pain when walking was unbearable and my most comfortable position was on my back. On the third day I saw a GP. I described my symptoms which included a fever, the shivers, the excruciating pain when walking and the fact that I had recently arrived from New Zealand. He had a brief look at the leg, asked a routine set of questions that I can’t remember now and wrote a letter, printed it and gave it to me with instructions to go to the A&E at Kingston Hospital. “Can I get some painkillers?” I asked. “Better find out what it is first” he replied. “Is it bad?” He paused for a bit and then said, “Maybe”. I left the GP and went to Kingston Hospital. Getting in and out of a car was an ordeal. I would first rotate my legs out of the car. Then holding onto the door frame for support I would haul myself up, using only my good leg for support and trying to minimize the bend in the left knee. I had a look at the letter the GP wrote. It said that I was to be treated immediately for suspected deep vein thrombosis (DVT). The part that said ‘treat immediately’ was a blessing. Those who have ever been to the A&E in London know that a 3 or 4 hour wait before being seen is common and I was glad I did not have to sit in the waiting room hours before being seen by a doctor.

The doctor at the A&E asked questions and I gave the same description of the symptoms. They did two blood tests, one from my vein and another from an artery in my wrist. Things seemed to get more serious as time progressed. It took an hour and a half for the results of the test to come back. The doctor said it was inconclusive for DVT although it was apparent that I had some sort of infection. My C-reactive protein (CRP) count was 450 mg/L. The CRP count is an indicator of infection or inflammation. A normal reading is less than 10 mg/L. My heart rate was beating at about 100 beats per minute so I had an ECG done. It showed no anomalies. The next surprise for me was the preventative anti-coagulant they administered via an injection to the stomach. It didn’t hurt. Then they put me into a bed and inserted a cannula and dripped the standard saline solution into me. By now I was wondering thoughts like Why are they doing this? Does this mean I get to stay the night?

I stayed the night. Shortly after being placed on a drip I was wheeled on my bed from the A&E to the medical assessment centre (MAC). By now it was late, about 20h00, and I was hungry. I had not had anything to eat since lunch and, because I was ill, I was not eating much anyway. In the MAC I was placed in a mixed bay with about 6 other patients, all elderly. One of them was a women who kept trying to take her gown off and the nurse, constantly had to tell her “There are men here, put your clothes back on”. She did this about 10 times. Later in the evening, at about 21h00, another doctor came to see me. He had a look at my leg under the bed light. “Looks like cellulitis in the left leg to me. I can see why the GP must have though DVT since you’ve arrived from a long flight.” I was placed on intravenous antibiotics for treating the bacteria causing the cellulitis in the leg. It was Tuesday the 27th of March 2007. The women in the bed next to me who kept trying to take off her clothes was moved from the ward. I think I must have dozed off but later woke a little later that night when someone else was wheeled in next to me. The next morning he was dead and in the afternoon they wheeled him out in a black body bag.

I think I stayed another night in the MAC before being moved to the Winter ward. This was a difficult journey because the lifts in the hospital, being somewhat prone to illness, stopped working. Before they could wheel me back, I made a little noise that was probably construed as a protest and said “I’ll walk the stairs.” Despite the obvious pain walking caused I knew I could pull of at least 1 flight of stairs. Grabbing onto the hand rail for support and being careful not to use my left leg for anything more than a temporary support I winced my way up the steps onto the next floor. From there they wheeled me in a chair to my new bed.

I knew I was in for a hard time in my new bed the moment I cornered the edge of the door. There were 6 beds in the bay, 3 on the left and 3 on the right. Mine was the middle left. To my right was an old man with no pants on lying in his bed and muttering and cursing to himself in a faint Irish accent. To my left was an old man sleeping so still that he could pass for dead. The whole bay was filled with old men. I must have been the youngest in the bay by 50 years.

It wasn’t long before the man on my right, called John, started yelling abuse at the nurses. This was a constant phenomenon for the 3 weeks I was next to him. I think they had removed his call bell because of the frequency he would use it and the constant abuse a well meaning nurse would receive from him. Often dialogue would go something like this:
“Someone help me up, help me up, help me up, help me up.”
Apparently he needed help to get out of bed so he could pee in his bottle. The ‘bottles’ were an thick egg cartoon material that males could pee into without having to walk to the toilet or even get out of bed. If no nurse arrived in time he’d pee in the bed. When the nurse arrived he would castigate them furiously saying,
“You brought this on yourself I tell you!”
On another grey London day a male nurse from Scotland was in the bay, working the IV and dispensing drugs to the patients. Unfortunately he managed to press John’s anger buttons, of which there must have been many, and John flamed at him
“Why don’t you go back to Scotland!”
John even had the balls to tell a chaplain to go to hell. She came to him offering communion.
“Hello John,” she said in her pleasant as custard and cake tone, “I’m the chaplain here to offer communion”.
“I don’t want communion from you,” he burst forth, “You’re a protestant and I’m a catholic!”
“Now John there’s no need for that”, the old man opposite him said, “It doesn’t matter.”
The chaplain was standing still flabbergasted by the verbal assault. I think she may have taken a step back in shock.
“Of course it matters,” he replied, “There’s a big difference. If it’s Catholic it’s the body and blood of Jesus otherwise it’s just bread and water.”
By now the chaplain had recovered some of her composure and she said, “I’m sorry you feel that way. I’ll go now.”
“That’s right. Piss off.”
I was sick of his yelling and belligerence after a few hours of being in the bay. When some friends came to see me on Wednesday we naturally started talking and laughing a bit.
I heard John start muttering something along the lines of, “There you are laughing and giggling away can’t you keep quiet!” I wasn’t going to let him get away with anything.
“Fuck you!” I yelled back.
I didn’t care if he was geriatric, if he wanted a conflict I wasn’t going to disappoint him. I yelled loud enough for people outside the bay to hear.
“Nobody likes you!”
That shut him up. When my friends and cousin left, John apologized to my cousin as he passed John’s bed. Although I did give him a dirty look when I passed his bed on the way to the toilet on one occasion, and yelled at him “You’re a weird man”, I wasn’t going to have a permanent war with him. Part of me realized that he was old and frail but his mind was still intact and I think it frustrated him not being able to do the things he would ordinarily do with ease. There was an occasion when he stood up and was looking for a bottle to pee in. He could not find one so I handed him one of mine. I also brought him some fresh water once when I walked passed his bed and he asked for it. However being old and frail didn’t excuse his rancid behaviour. There were other old people in the ward that behaved normally.

Being in hospital is a battle. You’re fighting several enemies. Firstly there is the illness that put you on a bed in the ward. This can range from something imaginary where a patient thinks he is sick and wants to be in hospital to or to something severe like a stroke. I saw a mixture of patients come and go while in the ward. The majority of the patients were elderly, 70 years and older. There was a patient with cellulitis in the leg. He stayed about 5 days before being discharged. Another patient came in after surgery. He had had about 24 cysts removed from his upper body and stayed about 2 days before his wife took him home. There was a 94 year old who had the mind of a 20 year old. He constantly hit on the nurses, saying things like “Come back to my place after I’m discharged. I’ll make you dinner and teach you French. I have French blood” and “The sight of you warms me up”. He lost his teeth one day and that caused some consternation amongst the occupational therapist and nurses. One day he walked around smiling at all the nurses until one of them pointed out that he had no pants on. He promptly sat on his bed and tried to use his shirt as pants. It wasn’t working out too well but fortuantely one of the nurses pointed out his mistake. It was always a laugh when whenever the nurses came to talk to him. Another patient came in and had a ‘fall’ next to his bed. He told the nurses and doctors that he could not walk. One morning I watched him as he got out of bed and shuffled sideways to his chair and sat down. However, for some reason, walking forward was beyond him. The nurses thought he was faking it and the doctors could detect no injury from his ‘fall’ that prevented him from walking. Another patient came because his GP called an ambulance for him. He managed to talk to everyone in the ward who would listen and he liked to emphasize the fact that he was there because his GP had called an ambulance and therefore his case must be serious. Apparently he was there because of the after effects of a gastric flu he had had 3 weeks ago. When he told me this I wished he wouldn’t sit so close and talk to me. The doctors didn’t treat him for the after effects of gastric flu because there was nothing to treat. Instead they diagnosed an anomaly in his blood tests resulting from the epilepsy medication that he had been taking for 10 years. He took great pains to tell everyone in the ward, nurses, doctors and pharmacists and anybody who would listen to him how dangerous it was changing the dosage and that it had to be done in a controlled way. He was discharged after about 2 or 3 days.

Secondly there is the boredom. A lot of time is spent staring at a ceiling or a wall. The most exciting times in the day are when the doctors do their rounds and visit the patients. This is when you get to talk to someone, tell them how you’re doing, hope that they’ll find out what’s wrong and fix things so you can go home. The other exciting times are meal times. You are presented a menu and get to choose one of three courses for dinner or lunch. I thought the food was quite good despite what everyone says about hospital food. Once they forgot to give us lunch, which made things even more exciting than usual. Other than wait for meals and talk to doctors and nurses there is not much to do. Visitors, people who like you, are a great source of support and are good at relieving the boredom. After 2 weeks in hospital my mother flew over from Australia and she stayed for the remainder of my stay. You can read if you have some books worth reading. It took me a while to find a book worth reading, but eventually I found The Silence of the Lambs and that entertained me for a day and a half.

Third is the constant disturbance from mad patients, like John next to me, who keep you awake at night with their moaning and yelling at nurses. It was 1 am in the morning when I heard a motion next to me. Standing with no pants on next to my bedside table was the man from the bed on my left. He was drinking from my water jug and spilling down his chest. I was a bit shocked and not knowing what to do told him,
“Go to the toilet”.
“Where is it?”
He’d been in the bay a few weeks and still didn’t know where the toilet was.
“That way”, I pointed in the direction of the toilet.
“Oh”, he said and shuffled along without trousers. The initial shock of him stealing my water at midnight made me wary of him for a few days but after a while I realized he was harmless funny old man. Later in my stay a patient with dementia arrived. He often forgot where his bed was and was inclined to wander over to other people’s beds and take their things. He tried to take some of John’s things from John’s precious bedside table and a verbal tornado of abuse ripped forth “Go away! Those are my things! Piss off back to your bed you useless thing!” After John left, which took 2.5 weeks, the bed on my right was filled with a man from Sri Lanka who was suffering from the after effects of a stroke. He could not stop talking and would babble loudly in Tamil about nothing. He did this all day and all night. None of the nurses could communicate with him. He could not move so he would go to the toilet in the bed resulting in a suffocating assault on my sense of smell and 3 nurses coming in to clean up the mess. One night, a nurse came into his bay to check on him.” “Oh god,” I heard the nurse murmur, “Oh god. Why did you do that? There’s blood everywhere.” The man had pulled out his catheter. A catheter is a flexible tube inserted into the urethra and allows one to urinate into an external bag. Pulling it out ruptured things internally causing him to bleed profusely from the urethra. He really was in a bad state.

On the 1st day being in the Winter ward one of the junior doctors came to assess my condition.
“Hi.” she said, “How do you feel?”
“There’s less pain in the leg,” I said, “although it is still swollen. The redness seems to have gone. It still hurts to walk.”
“That’s good. Looks like the antibiotics are working.”
A few minutes later her registrar, a more senior doctor, came in to follow up. He had a look at the leg and my temperature chart and said, “We need a scan done on the leg just to make sure. Also his temperature is spiking. Double his antibiotics.”
I’m not sure if I heard the next part correctly or not but I think he said “And scratch that ‘recommend for discharge’ note”. I may have misheard but it was somewhat disconcerting to think that the junior doctor had missed the temperature spiking and possibly recommended that I be discharged when I could not walk because of the pain.

I think by now I was on 1.2g of benzyl penicillin and 1.0g of flucloxacillin administered intravenously. The next day, Friday, I was scheduled for an ultrasound on the deformed leg. I was placed on a wheelchair and given a blanket to wrap myself up with. It was cold between wards and the trip to the radiographer’s lab was long. Because I could not bend my knee fully I had to keep it away from the footrest on the chair. There may have been a light snow outside. The radiographer scanned my leg with the transducer looking for DVT. She moved the transducer over the back of the knee. “Look at this” she said to the other radiographer in the room. The other radiographer moved to look at the image on the scanner. “What is it?” I asked. There was a pause while they both looked at the image on the screen. “It’s fluid in the knee. That’s why you’re in so much pain.”

Armed with this new knowledge and the feeling that I had finally found someone who understood the pain I was in, I went back to bed and waited. It was Friday. The doctors had already done their rounds in the morning. The next day would be boring because the doctors do not do rounds on the weekend. I would have to wait till Monday for the next stage in the diagnosis. I played on the PSP. My cousin had bought this neat wrestling game, Smackdown vs. RAW, that I ended up playing to the tournament’s end. It had a massive variety of ways to customize a player created wrestler, from the introductory walk into the arena to the attire and allowed the player to assign from hundreds of moves, like backbreakers, suplexes, punches and chops, kicks and mudhole stomps.

When the pain in the knee was too great I could ask for tramadol or dihydrocodeine. They are both painkillers used to treat moderate to severe pain and they helped relieve the pain present in the knee when it was not bent beyond its limits. My guess is they are better at relieving visceral pain rather than joint and muscular pain. There was nothing to relieve the pain if I did try to bend it beyond the limits imposed on it by the fluid build up. By Saturday afternoon I felt more pain in the knee and asked for painkillers. They brought a welcome relief. I think the nice nurse gave me a combination of tramadol and dihydrocodeine which was awesome. I found that I could limp my way to the toilet without wincing and for a while I was happy.

Midnight came and I was hurting badly. The effects of the painkillers had worn off. I couldn’t sleep because of the pain in the knee. I wish it would go away. Maybe if I wait a bit it will go away. I waited till 2 am still not sleeping and tossing and turning trying to find the least painful position for the knee. Eventually I gave up trying to get through the night on my own and called the nurse.
“What’s wrong,” she asked.
“My knee. It’s more swollen and it hurts. I’m in pain. Maybe call the doctor.”
“If I call the doctor he’ll recommend a painkiller. So what I’ll do is give you a painkiller and if the pain is still there then we’ll call the doctor, ok?”
She gave me two pills of dihydrocodeine and left. About an hour later I was still hurting the painkillers having no effect. I pressed the call button again. This time a different nurse came in, one who was not allowed to administer drugs.
“What’s wrong?” she asked.
“I’m in pain. My knee hurts. I just want it to go away.” My voice was weak and barely above a whisper.
“I’ll call the other nurse,” she said. As she walked away she asked stupidly “So the painkillers aren’t working?”
“No!” I yelled back angrily.
Sometimes the stupidity of people and their ability to ignore the obvious was infuriating.

It was about 3 am on Sunday morning when the doctor came into the bay. He turned on the bed light and proceeded examining me legs and knee. I told him where the pain was. Then he began tapping a point high on my right leg and the mirrored point on the left leg.
“Does it feel the same?” he asked.
“The same,” I replied. He then tapped a little lower. “And now. What does it feel like?”
“The same,” I replied.
I’m guessing he was looking for some sort of failure in my nervous system. I thought this was a stupid test given that the size of my left knee was 3 times the size of my good right knee. It was pretty obvious that the pain was real, located in my left knee, and not a result of a malfunctioning nervous system. He continued tapping away until he reached my feet and then left. The next thing I remember was an orthopaedic surgeon at my bed. It was about 4 am now. He looked at the knee.
“I didn’t think they would have an orthopaedic surgeon around at this time.” I said in a vaguely impressed tone.
“They keep me here.” he said. “You’re from New Zealand?” “Yes.” I replied.
“I’m half Australian so I can tell the difference between the accents. How much can you bend the knee?”
I tried to bend it but couldn’t. It was locked at a certain angle.
“I’m going to twist your leg to see where the source of the pain is. Have to make sure.”
So he began twisting it from left to right. Once he passed the median point, the point where it begins to twist inwards, I yelled as a sharp pain lanced through the knee.
“Don’t do that!”
“Ok, sorry, but I have to find out where the pain is. I’m going to come back with a syringe and drain the knee.”
He returned and asked me to bring my leg up so that there was an arch between the bed and the knee with the knee at the apex of the arch.
“I’m going to anaesthetize the knee. It will hurt.”
The needle went in from the outer left side of the knee. I clutched the sides of the bed and screamed. Next he pulled out a larger 50ml syringe attached to a needle with the largest bore I’d ever seen. I had a brief flutter of panic. He inserted it into the knee and the syringe rapidly began to fill with yellow fluid. The relief from the pain was instant. I could breathe again. He filled a 2nd syringe with more synovial fluid and moved onto a 3rd. He stopped when blood started to enter the syringe.
“Looks like you have arthritis,” he said.
I asked. “I’m too young for that.”
I was puzzled because I had only ever heard of arthritis in old people.
“Not really,” he replied. “We’ll get the doctors to have a look at you early in the morning.”
I slept instantly.

It was about 9 am and a team of about 5 doctors had swarmed into my bed area. The curtains were drawn. A tall man, with black hair and bald on the top, took control of the situation.
“Hello. What was the first thing you noticed when you got sick?”
“I woke up shivering. Moving the leg and knee was painful.”
“So you noticed the knee first?” he asked.
“It was one of the first things,” I replied.
“Do you know why the doctors are worried about you?” he asked.
“Because the current treatment is not working.”
“Yes. Well it looks like a case of septic arthritis. Being the most senior doctor here,” he said, “We’ll schedule you for surgery this afternoon and open up the knee and wash out the fluid. It should get rid of most of the bacteria, if not all of it. Hopefully a theatre will be free.”
It finally looked like the doctors had found, after 6, days a correct diagnosis. From then onwards I was dealt with by the orthopaedics team and no longer the team from general medicine.

There’s not much to say about surgery because I was under general anaesthetic. After the surgery my knee felt the most comfortable it had felt in a long time. It was wrapped in a pressure bandage. I was surprised by the lack of pain. I expected there to be pain from the surgery. I was wheeled from the recovery room, where you swim through double vision while waking up from the anaesthetic, back to my bed. Another surgeon came to speak to me after the surgery to make sure everything was ok.
“Well what we found in the knee was not normal. There was a build up of pus there,” he said, “but hopefully we’ve washed out the bacteria. We put through 6 litres of fluid and sent the extracted fluid to the microbiologists to see if they can grow the bacteria. We’ve also placed you on a great whacking dose of antibiotics.”
I later found out the amount I was on. The nurses administered the antibiotics by mixing it in a sack of saline solution. Then they labelled it with a yellow label on the sack which stated the name of the antibiotic and the amount and connected it to the cannula in my vein. I was on 2.4 grams of benzyl penicillin and 2.0 grams of flucloxacillin for about 2.5 weeks every 4 hours. Apparently this is a large dose. I had another knee arthroscopy with a wash out 5 days later on Friday. This time they did not find any pus in the fluid build up, however there was a lot of internal bleeding in the knee from the last operation. The pressure bandage after that operation was soaked with blood and it coagulated freezing the bandage solid.

After they removed the pressure bandage the knee was still swollen. The swelling was an after effect of the infection and it had affected the ligaments. They tried different antibiotics in order to improve the results from my blood test. Eventually they settled on a combination of flucloxacillin, penicillin and the ‘drug of last resort’ ciprofloxacin. The goal was to get my CRP count down to acceptable levels. The length of my stay in hospital seemed to be based on my CRP count as well as improving mobility in the knee. I moved around on crutches and was surprised by how weak I had become. I was given physio and exercises to do to regain mobility and strength in the knee. By the end of the third week the nightly sweats I had been having had stopped. My temperature had stabilized and I was no longer spiking. By the time of discharge, after 4 weeks, my quadriceps and calf muscles had withered away noticeably. I’d lost about 8 kilos in the 4 weeks I’d been in hospital since falling ill. A few days after leaving hospital I found that I could walk with a limp without the aid of crutches. Walking for long periods was more tiring than usual. As of this writing it’s been about 6 weeks since I fell ill and 5 weeks since the first knee arthroscopy. There is still a slight, steadily diminishing, swelling around the top of the knee and I probably have about 90% movement in the knee.

Mergesort in Julia

I became interested in learning more about Julia because it was

  • based on LLVM (none of this JVM/Java bullshit)
  • supports first class functions
  • support for parallelization of tasks over multiple cores or nodes is built into the language
  • it is influenced by the numerical/mathematical programming languages such as R, Matlab and Fortran and maybe I want to write simulations again (wishful thinking).

so in the tradition of not many other first programs in a new language I wrote a merge sort routine

It doesn’t use any of the features of the language I’m interested in, however:

  • I did it in about an hour and a half.
  • I learnt to use 1 as the base index offset after years of using 0.
  • Someone more versed in Julia could probably rewrite the combine part of the merge sort using fancy zips and maps
  • I learnt not to expect 1/2 to return 0.

cbf fish

Can’t be fucked using fish anymore. The subtle incompatibilities with bash waste my time. For example this won’t work:

find . -iname "*.java" -exec grep -in "bond" {} \;

Creating schroots

Chroot jails are a neat way of building for target Linux distros without having to run the distro in a vm. For example, in my case, I can host the filesystem for RHEL on my Arch linux machine and then build binaries for RHEL that will be compatible. All I need to install is rpm and yum and because it’s not native to Arch which I got from the Arch user repository. A program call schroot makes the configuration and mounting the chroot environment easier and less error prone. It will mount /dev and /proc and /home from the host into the chroot and copy basic system database files over so that when you log into the chroot your home directory from the host is automatically mounted inside the chroot guest and you’ll be able to use the system like it is a native (insert name of guest chroot here) distribution.


This is how I setup the RHEL chroot on Arch. Obviously I installed schroot, rpm and yum previously. How you install those programs depends on your distro.

sudo mkdir /srv/chroot/rhel
wget http://<some yum repo>/packages/redhat-release-server-6Server-
sudo rpm --root=/srv/chroot/rhel --nodeps -i redhat-release-server-6Server-

Then configure yum on the chroot to point to the yum repository you want to use. For example:

$ cat /srv/chroot/rhel/etc/yum.repos.d/rhel-ericsson.repo

Then install yum in the chroot.

sudo yum --installroot=/srv/chroot/rhel6.6/ install -y yum

Now configure schroot for this guest. Create an entry like the following in /etc/schroot.conf.

description=RHEL 6.6

The keys are fairly self-explanatory. The type is directory because well the guest is layed out in, yes you guessed it, a directory /srv/chroot/rhel. users are the users you want to allow access to the chroot. root-users are users on the host you want to allow to login as root inside the guest. profile is the name of the schroot profile described next. See man schroot.conf for a full description of all keys.

Next create a directory with the same name assigned to profile.

sudo mkdir /etc/schroot/rhel

Add the following files in that directory. For example:

sashan@phoenix ~/code  > cat /etc/schroot/rhel/nssdatabases
# System databases to copy into the chroot from the host system.
# <database name>
sashan@phoenix ~/code  >
sashan@phoenix ~/code  > cat /etc/schroot/rhel/copyfiles
# Files to copy into the chroot from the host system.
# <source and destination>
sashan@phoenix ~/code  > cat /etc/schroot/rhel/fstab
# fstab: static file system information for chroots.
# Note that the mount point will be prefixed by the chroot path
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/proc           /proc           none    rw,bind         0       0
/sys            /sys            none    rw,bind         0       0
/dev            /dev            none    rw,bind         0       0
/dev/pts        /dev/pts        none    rw,bind         0       0
/home           /home           none    rw,bind         0       0
/tmp            /tmp            none    rw,bind         0       0

sashan@phoenix ~/code  >

Then you are done. Now you can login to the chroot as your username or root. For example:

schroot -u sashan -l rhel -s /bin/bash


schroot -u root -l rhel -s /bin/bash

Once inside you might need to run

$ rpm -vv --rebuilddb

to make sure the rpm database works nicely.

A new OS

I was investigating cloud foundry because I wanted to see how it differed from OpenSAF. I watched their presentation and posted a question to their mailing list. It hasn’t arrived yet and it’s been 24 hours. And then I wanted to install a private cloud foundry instance. It looks like I should be able to do it. But then I realized that it required something to run multiple virtual machines and I was running linux in a vm on windows. Not keen to run a vm inside a vm or run multiple machines from my Windows host because I can’t be bothered using Windows as development environment anymore, I decided to pull out an 8 year old hard drive from a dead computer lying around and stuck it into my 2.5 year old desktop machine, installed Arch on it, decided to use i3 and not Gnome, found some cool colorschemes from, wrote a script to randomly pick a colorscheme when I open a new urxvt terminal via mod+enter in i3, installed the nvidia drivers so I can do some coding on the GTX680 inside, discovered the Red Screen of Death when using nvidia-libgl, fixed RSOD, found out that my Cougar 700K gaming keyboard literally halts when I press any of the special keys on it and I have to reset it by unplugging it from the USB port, found out that my ASUS Phoebus gaming sound card doesn’t work at all in Linux and my gazillion button gaming R.A.T mouse works like a 3 button mouse. I named my new Linux machine Phoenix because it rose from the derelict remains of an 8 year old disk drive. And then the weekend ended along with my cloud foundry investigation.



I started using the Friendly Interactive Shell. Here’s a screenshot showing the git branch and error code of the last run program at the command prompt.

Code is on github