How to write MCollective Agents to run actions in background

The Marionette Collective, or MCollective in short, is a cutting-edge tech for running system administration tasks and jobs in parallel against a cluster of servers. When the number of servers you have to manage grows, the task of managing them, including keeping the OS and packages installed on them updated, becomes without a doubt a nightmare. MCollective helps you drag yourself out of that nightmare and into a jolly dream, where you are the king, and at your disposal is a powerful tool, by merely wielding which you can control all of your servers in one go. I’ll probably do a bad job of painting MCollective in good light, so I’d recommend you read all about it on MCollective’s main website.

Like every good tool worth its while, MCollective gives you the power to extend it by writing what it calls agents to run custom code on your servers to perform any kind of job. You can read about agents here. MCollective is all in Ruby, so if you know Ruby, which is a pretty little programming language by the way, you can take full advantage of MCollective. Incidentally, a part of my day job revolves around writing MCollective agents to automate all sorts of jobs you can think of performing on a server.

For a while I have been perplexed at the lack of support for being able to run agents in the background. Not every job takes milliseconds to finish itself. Most average-level jobs, in terms of what they have to do, take anywhere from seconds to, even longer, minutes. And since I write an API which uses MCollective agents to execute jobs, I often run into the problem of having the API block while the agent is taking its sweet time to run. As far as I’ve looked, I haven’t found support within MCollective for asynchronously running actions.

So, I got down to thinking, and came up with a solution, which you could call a bit of a hack. But insofar as my experience testing it has been, I’ve yet to face any issues with it.

I’ve an example hosted on GitHub. It’s very straightforward, even if crude. The code is self explanatory. At the heart of it is creating your actions in agents to fork and run as childs, without having the parent wait for the childs to reap, and having one extra action for each agent to fetch the status of the other actions. So, essentially, the approach has to be taken for every agent you have, but only for those actions which you wish to run asynchronously. With agents in place, all you’ll need to do is call the agents thus:

$ mco rpc bg run_bg -I node -j
{"status": 0, "result": "2434"}

and repeatedly fetch the status of the async action thus:

$ mco rpc bg status pid=2434 operation=run_bug -I node -j
{"status": 0, "result": ""}

It’s a solution that works. I’ve tested it over a month and a half now over many servers without any issues. I would like to see people play with it. The code is full of comments which explain how to do what. But if you have any questions, I’d love to entertain them

Fastly’s CDN accessible once again from Pakistan.

Three days ago I wrote about a particular subnet of Fastly.com’s CDN network becoming null-routed in Pakistan. Since the affected subnet was from which Fastly served content to users from Pakistan, websites, such as GitHub, Foursquare, The Guardian, Apache Maven, The Verge, Gawker, Wikia, even Urban Dictionary and many others, which off-load their content to Fastly’s CDN networks stopped opening for users inside Pakistan.

However, as of today, I can see that the null-route previously in place has been lifted as mysteriously as it was placed. The subnet in question, 185.31.17.0/24, is once again accessible. I have tested from behind both TransWorld and PTCL. While I don’t know why it was blocked in the first place and why it has been made accessible again, whether it was due to an ignorant glitch on someone’s part, or whether it was intentional, I am glad that the CDN is visible again.

If you are observing evidence otherwise, please feel free to let me know.

185.31.17.0/24 subnet of Fastly’s CDN null-routed in Pakistan?

I rely heavily on GitHub and Foursquare every day, the former for work and pleasure, and the latter for keeping a track of where I go through the course of a day. Since yesterday, though, I have been noticing that pages on GitHub have been taking close to an eternity to open, if not completely failing. Even when the page loads, all of the static content is missing, and many other things aren’t working. With FourSquare, I haven’t been able to get a list of places to check in to. Yesterday, I wrote them off as glitches on both Foursquare and GitHub’s network.
 
It was only today that I realized what’s going on. GitHub and Foursquare both rely on Fastly’s CDN services. And, for some reason, Fastly’s CDN services have not been working within Pakistan.
 
The first thing I did was look up Fastly’s website and found that it didn’t open for me. Whoa! GitHub’s not working, Foursquare’s not loading, and now, I can’t get to Fastly.
 
I ran a traceroute to Fastly, and to my utter surprise, the trace ended up with a !X (comms administratively prohibited) response from one of the level3.net routers.
 
$ traceroute fastly.com
traceroute: Warning: fastly.com has multiple addresses; using 216.146.46.10
traceroute to fastly.com (216.146.46.10), 64 hops max, 52 byte packets
[...]
 6 xe-8-1-3.edge4.frankfurt1.level3.net (212.162.25.89) 157.577 ms 158.102 ms 166.088 ms
 7 vlan80.csw3.frankfurt1.level3.net (4.69.154.190) 236.032 ms
 vlan60.csw1.frankfurt1.level3.net (4.69.154.62) 236.247 ms 236.731 ms
 8 ae-72-72.ebr2.frankfurt1.level3.net (4.69.140.21) 236.029 ms 236.606 ms
 ae-62-62.ebr2.frankfurt1.level3.net (4.69.140.17) 236.804 ms
 9 ae-22-22.ebr2.london1.level3.net (4.69.148.189) 236.159 ms
 ae-24-24.ebr2.london1.level3.net (4.69.148.197) 236.017 ms
 ae-23-23.ebr2.london1.level3.net (4.69.148.193) 236.115 ms
10 ae-42-42.ebr1.newyork1.level3.net (4.69.137.70) 235.838 ms
 ae-41-41.ebr1.newyork1.level3.net (4.69.137.66) 236.237 ms
 ae-43-43.ebr1.newyork1.level3.net (4.69.137.74) 235.998 ms
11 ae-91-91.csw4.newyork1.level3.net (4.69.134.78) 235.980 ms
 ae-81-81.csw3.newyork1.level3.net (4.69.134.74) 236.211 ms 235.548 ms
12 ae-23-70.car3.newyork1.level3.net (4.69.155.69) 236.151 ms 235.730 ms
 ae-43-90.car3.newyork1.level3.net (4.69.155.197) 235.768 ms
13 dynamic-net.car3.newyork1.level3.net (4.53.90.150) 236.116 ms 236.453 ms 236.565 ms
14 dynamic-net.car3.newyork1.level3.net (4.53.90.150) 237.399 ms !X 236.225 ms !X 235.870 ms !X

Now, that, I thought, was most odd. Why was level3 prohibiting the trace?

I went looking for a support contact at Fastly to try and get anything that could explain what was going on. I found their IRC chat room on FreeNode (I spend a lot of time on FreeNode), and didn’t waste time dropping into it. The kind folks there told me that they’d had reports of one of their IP ranges being null-blocked in Pakistan. It was the 185.31.17.0/24 range. I did some network prodding about, and confirmed that that indeed was the subnet I couldn’t get to from within Pakistan.

$ ping -c 1 185.31.18.133
PING 185.31.18.133 (185.31.18.133): 56 data bytes
64 bytes from 185.31.18.133: icmp_seq=0 ttl=55 time=145.194 ms
--- 185.31.18.133 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 145.194/145.194/145.194/0.000 ms

$ ping -c 1 185.31.16.133
PING 185.31.16.133 (185.31.16.133): 56 data bytes
64 bytes from 185.31.16.133: icmp_seq=0 ttl=51 time=188.872 ms
--- 185.31.16.133 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 188.872/188.872/188.872/0.000 ms

$ ping -c 1 185.31.17.133
PING 185.31.17.133 (185.31.17.133): 56 data bytes
--- 185.31.17.133 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss

They also told me they’d had reports of both PTCL and TWA where the range in question was null-routed. They said they didn’t know why it had been null-routed but would appreciate any info the locals could provide.

This is ludicrous. After Wi-Tribe filtering UDP DNS packets to Google’s DNS and OpenDNS servers (which they still do), this is the second absolutely preposterous thing that has pissed me off.

Docker

Docker is almost the rage these days. If you don’t know what Docker is, you should head on over www.docker.io. It’s a container engine, designed to run on Virtual Machines, on bare-metal physical servers, on OpenStack clusters, on AWS instances, or pretty much any other form of machine incarnation you could think of. With Docker, you can easily create very lightweight containers that run your applications. What’s really stunning about Docker is that you can create a lightweight container for your application in your development environment once, and have the same container run at scale in a production environment.

The best way to appreciate the power Docker puts at your fingertips is to try it out for yourself. If you wish to do it, I would recommend the browser-based interactive tutorial on Docker’s website.

While it is easy to build Docker containers manually, the real power of Docker comes from what is known as a Dockerfile. A Dockerfile, using a very simple syntax that can be learnt quickly, makes possible automating the process of setting up and configuring the container’s environment to run your application.

On a weekend I finally took out time for myself and sat down to embrace Docker, not only through the interactive tutorial on Docker’s website, but also on my server. I was half lucky in that I didn’t need to have to set Docker up on my local system or VM, because Linode just happened to very recently introduce support for Docker. I started playing around with Docker commands on the shell manually, and slowly transitioned to writing my own Dockerfile. The result: I wrote a Dockerfile to run a small container to run “irssi” inside it. Go ahead and check it out, please. If you have a system with Docker running on it (and I really think you should have one), you can follow the two or three commands listed on the README file to build and run my container. It is that easy!

iPad mini retina image retention

I first read about image retention on iPad mini retina on Marco Arment’s blog. His iPad mini retina had a slight case of image retention which he discovered by [creating and] running an image retention test on his iPad. I used the word slight to describe Marco’s case because his was a minor problem, something had he not run the test explicitly would not have noticed during normal use. Because it didn’t seem something that would get in the way of enjoying the beautiful screen of the new iPad mini, I didn’t give it much thought.

The very first thing, after hooking it up online, I did on my new iPad mini retina was run Marco’s image retention test. It passed, which elated me and squashed what little fears I had. In hindsight I forgot to run the test for ten minutes, hastily choosing a minute instead. I basked in the magnificence of the retina screen and the weightlessness of the device for two whole weeks. It was the perfect tablet: light-weight, just the right size, with a beautifully sharp and crisp screen, a lot of computing power packed inside a small form factor, and a lovely OS to make it all work seamlessly. Then, one unfortunate night after work when I pulled out the iPad from inside the drawer I keep it in when away, I was dreadfully shocked to look at the mess the screen had become. The image retention clearly visible on the screen was horrible. There were crooked lines everywhere, and swiping on the screen caused them to flicker grotesquely. If Marco saw it, he would jump up his chair.

Home screen on my iPad mini with severe image retention.

Home screen on my iPad mini with severe image retention.

I managed to get the iPad returned to Apple. To my surprise, and a little disappointment, Apple outright refunded me the device.

Macro in his post explained why he thought the issue was there. Apple buys retina panels from a couple of manufacturers. Panels from at least one manufacturer exhibit image retention. I think Apple is fully aware of it, and it’s the reason why iPad mini with retina displays are in short supply.

I loved that thing. I cannot emphasise that enough. I will buy it again, when the next batch from manufacturing hits the market.

→ Don’t ask a non-drinker why they don’t drink

storm_cloaks, commenting on a Reddit thread:

As a non-drinker, answering the “why don’t you drink?” question is always annoying. Generally speaking, I think it’s poor etiquette to ask someone why they don’t drink, and it saddens me that most people don’t feel the same. First, its really none of their business. Second, asking someone to justify a personal choice at a party is a total killjoy, and it clearly creates a separation between the non-drinker and the drinker that’s asking. Going out of your way to point out the fact that someone’s different from you, *especially in a situation that’s supposed to be festive is totally ridiculous, if not offensive. I understand the curiosity, but it would be rude and odd if I asked people at a party “why do you drink?”

I love that explanation. I couldn’t have put it any better myself. On several occasions I have been asked by friends and acquaintances why I don’t drink. 90% of the those times, I have been made fun of and called a pussy for not drinking. I have never understood why people who drink or want to drink feel compelled to ask why the people around them don’t. It’s OK to offer somebody a drink, but not at all OK to ask them why they don’t. It is OK if you are truly curious why somebody doesn’t drink, but I feel that people who drink don’t honestly care about the reason why somebody doesn’t drink. I mean, are they looking for reasons to justify quitting? Or looking for lack of justifiable reasons to justify their drinking?

Change, and adversity, and a goodbye to 2013!

Early morning WordPress sent me that email they send once every year underlining all the special stats about your blog from the year that is about to pass. I didn’t open the email until very late in the day. I was shocked to see that I had written and published only two posts the entire year. I thought that was a rather sad and unfortunate state of affairs. I love writing. I wrote a lot what would be safe to call once upon a time. To think, no, to know for a fact that things had gone this bad, shook me.

That in part explains why I felt compelled at the very last minutes before the year rolled over to write this. I am going to try and see this till the finish.

If I were to describe this year in two words, I would carefully choose those words to be: “change” and “adversity”. I think those two words are not only powerful, but also the kind that exude meaning. They describe the kind of experiences you read about in books or in people’s anecdotes or watch in movies, and feel a special force moving you from within. You feel compelled to take in the moment, and while you are reading or watching those experiences, you play them in your head around you, if only to know how it would feel to live those experiences.

Someone once said I’m risk-averse. It’s economic lingo. But it is also true. I don’t take risks unnecessarily, and I prefer to not take risks at all. It’s nothing wrong. A lot of people are risk-averse. A lot of people detest change. I think it is largely due to our personal inertia that we exert on life as we live through life. It takes a great amount of force to get oneself to embrace change. For most people, it is easier to just resist change.

Uncharacteristically, I embraced several changes this year. Some were huge changes for me. Some, not so. But none were any less meaningful than the rest.

After working four and a half years from home, I decided, I don’t know how, to quit. Quitting from anything is a monstrous challenge for me. It always has been for as long as I have remembered to notice it. Sure, I feel the urge to run away from things from time to time, but quitting comes hard to me.

Not only did I quit, but I also got myself a job at a place where I had to go every day to an office and work from there. I hadn’t worked in an office in under five years. I didn’t know whether I could do it — if I’d be able to do it. I didn’t know whether I could survive it long enough. Unless you have experienced both, you cannot know how entirely different working from home is from working from an office. You have to get up on time in the morning, dress up semi-properly (if you want, you could just as well work in your underpants from home, you know), and commute to work. Oh yes, there’s always the long commute in rush hour to dread every day. You have to spend eight to nine hours straight in the office where you can’t take a nap or watch TV to take a long break away from everything. All in all, you lose a great deal of freedoms that you enjoy working from home. And who wants to give up their freedoms. The majority of people who found out that I worked from home thought it was a luxury I was lucky enough to afford. And I agree. Not everyone can afford to work from home. You are really lucky if you can. Despite all that, I decided to quit. In spite of all that, I quit. I quit because over four years is a long time. I wanted change. I wanted change in my routine, the kind of change I couldn’t bring about by altering anything else in my life without giving up working from home. I felt my life had become stagnant and that I wasn’t moving ahead at all. In fact, to the contrary, I felt I was being sucked into a dark, depressing place, and it frustrated me and caused me uncontrollable anxiety. I had to change things. I had to quit. And so, just like that, I did.

It has been seven months since quitting home job. I’m still working from the same office I get up every morning to drive to.

I changed tennis clubs. If you know me personally or semi-personally, you’ll know that I am crazy about playing tennis. I would go as far as to say that tennis is the love of my life. I started playing tennis four years ago at a country club which housed two shabbily built hard courts in the open. There was a rather dejected and disgruntled squash-cum-tennis coach in the club who helped me get on my feet. As I transitioned past the amateur level, I signed up in the local tennis circuits and started playing tennis tournaments. I earned a worthwhile ranking for a player like myself. However, the only place I could practice was the country club. And that club had major issues that got in the way of my practising and improving my game. I will keep myself from going into those, but I will say that they boiled up to a point where they started to really frustrate me and hamper my ability to excel. On top of that, there were hardly any players to play against at the club, much less good players. You cannot improve until you can hit with a better player.

I had been eyeing a professional tennis club for a while, but several fears that I molded  as convenient excuses kept me from making the jump. That club was far away. I didn’t think I could take a break that long to drive all the way and back. I was afraid of all the new people there. I was afraid whether I’d be able to find anyone to play with. In hindsight, it was mostly only fear of taking on a big change. It would’ve been big for me.

We always need that impetus, that push, to jump off the plank and see what’s out there. My changing jobs was that push. Incidentally, the office I was going to join was closer to that professional tennis club. I quit my previous club the very first day I joined the new office. I am glad I did.

I sold my car and bought a new one. For a lot of people, selling and buying cars is trivial. But normal people buy and sell cars once in a while. After all, my dad still owns and drives his two really old cars. I bought my very own car four years ago, brand new. I loved it. I love and treasure all of my possessions. However, three years later I felt I needed to upgrade it, and get something more comfortable. But selling a car felt so much of a nuisance. There was inertia kicking in all over again. Even if I did sell it, I thought, looking for another car to buy would be another troublesome affair I didn’t want to go through. I needed another push. And it came a year and a half later.

My changing jobs and my changing tennis clubs became that impetus. When I look closely at how things often unfold, it amuses me. I wonder whether there really is a time and place for everything. And for change. But that is a topic for philosophical musings, and I shouldn’t get sucked into that since I’ve promised myself to see this piece through to the end.

Adversity tests you and the people around you in ways you cannot consciously imagine otherwise. It shows you how weak and frail you are, and it shows you how strong and resilient you can be. It shows you who you thought didn’t love you really loves you, and it shows you who you thought didn’t care much about you actually cares for you more than others. Adversity is like that pair of glasses which when you wear them strips everybody including yourself off of their layers and facades, and lays them bare, in their true forms, for you to see.

My father fell acutely ill several times in succession later this year. He’s a diabetic of thirty long years, a heart patient with several allergies and a heart doctor himself. We had to rush him into emergency twice. He had to stay in critical care units for a total of twenty or so days, at the end of which he was diagnosed with two main blocked arteries, with his heart unable to pump blood into half of his body. He nearly kissed death, and survived. It was the toughest time of my life, and the hardest thing I had to deal with. I wrote a painful account of it but only got myself to publish it anonymously. I’m sorry but I couldn’t share it here.

I am thankful deeply to the few people who were around when I needed help and support the most. And I am thankful to God for being able to withstand and overcome adversity. It wasn’t easy.

When the year started, I set myself up for a 2013 reading challenge on Goodreads. I set myself an achievable goal of ten books. To my utter surprise, I crossed that goal in the next three months. I had to re-arrange the goal to 30 books instead. However, as I close off the year, it is only with sadness that I try to reconcile the fact that I only managed to read a total of fifteen books over the year. That is fifty percent of the goal. I finished the fifteenth book a night before new year’s eve. It was Mark Haddon’s incredibly cute and creative “The curious incident of the dog in the night-time.” It of course comes highly recommended. If you’d like to read about the other fourteen books I read, you could look them up here: My Goodreads 2013 Reading Challenge.

One of my gripes with the passing year is that I couldn’t play much tennis, despite switching tennis clubs. I also couldn’t play any tournaments. Owing to the office job, I cannot take out time during the afternoon and day to play tournaments. And because I don’t have so much freedom now as I did when I worked from home, I can’t take a break from work and drive by the courts to play, which is the reason I couldn’t play a lot of tennis this year. It is something I dearly wish to change in the coming year. Because, after all, not being able to play tennis is a cause of much panic and frustration for me.

I have dispensed with the little things while writing this account of the year. I only focused on what I felt were big things for me that shaped my life in 2013. And, also, I had to finish this, and I couldn’t allow it to stretch any longer, because that would risk giving me an excuse to not see it through the end.

My best wishes for 2014!