rsync-ing only dotdirs and dotfiles (hidden directories and files on unix)
TL;DR:
ls [source] | rsync -a –exclude-from=- [source] [dest]
I’ve recently been working on setting up a backup solution for my various computers. One of the things I wanted to do was backup only my program settings. In unix-like operating systems, these are usually stored in the user’s home directory, with a . in the beginning of the filename. This denotes that the file is hidden by default.
Rsync is a utility that is used to copy files from a source to a destination, optionally over a network. It is great for backups since it checks the contents of the files on both sides, and only sends the (parts of) the files that have changed.
Rsync can be a bit cumbersome to use, and I searched google for a while and even asked in their IRC channel for help. I couldn’t figure out how to make rsync select only my dotdirs. I figured it out today.
The command is:
ls [source] | rsync -a –exclude-from=- [source] [dest]
How it works: by default, ls only shows non-hidden files. So, the command ls builds a list of files and directories that are not hidden - a list of everything I don’t want. I pipe that to rsync -a (a is for archive). –exclude-from usually operates on files, but when you give it a - as a filename, it reads from stdin instead. In this case, stdin is the listing of stuff that I don’t want, piped from ls. Works perfectly.
Comments (0)File managers, parallel copy, and performance hits
Scenario:
I’m using an everyday modern GUI file manager to copy a huge file to my portable harddrive. A few minutes later, while the first transfer is still in progress, I realize that there were actually two huge files I want to copy to my portable harddrive.
What are my options? I could copy the other file right away, or I can babysit the file copy and copy the second file when the first one is complete.
The disadvantage to the first option is that the files copy in parallel. Harddrives operate best when used sequentially. Copying the files in parallel reduces the total rate at which data is copied to the harddrive, thanks to the head having to jump around to different parts of the disk. The total time of the operation is significantly increased.
The disadvantage to the second option is that I have to sit here and babysit a process that should be automatic. I’m just copying two files. What if my two files would take 15 minutes each to copy, but I want to do something that is going to take half an hour, and then come back and collect my portable harddrive and then leave immediately?
Another similar scenario: I want to copy two huge files to the same destination, but they’re in different source directories, meaning I can’t select both of them at once.
The solution? I think a queue would work nicely, but while solving one problem would introduce several others:
- User confusion: “why aren’t my files copying [right away]?”
- Having to add a complex UI to otherwise simple and unintrusive file-copy dialogs to handle queues - at the very least, you need to be able to remove the items from a queue
- Priority/detection of when queues should be used; should queues always be used when the destination device is the same for every file? Should queuing be explicit [Copy -> Paste (add to queue)]? Or maybe it is parallel that should be explicit [Copy->Paste Immediately]?
Why don’t we see a queue feature in modern file managers? Is it because it’s just a hard problem, or is there a good reason not to have it? Or perhaps I’m just using the wrong file manager?
Comments (24)Pro Starcraft In English - A RSS Feed
I’ve picked up starcraft fever again. And, I’ve started watching professional games out of Korea.
For all of us non-Koreans, we’re able to get the videos in VOD form; that is, someone records the match from TV and posts it to the internet for the rest of us. However, these matches are almost always commentated in Korean. They’re still exciting to watch, but the commentary adds a lot to understanding why the players are playing how they’re playing.
Fortunately, GOM TV has started to provide internet-only english coverage of their tournament with tasteless, an American starcraft progamer. This is available in both a live format and as a series of recorded videos after the match (VOD). Also, there are a number of excellent amateur commentators that provide english commentary over top of Korean VODs, from sc2gg.com.
Obviously, I want to follow these games, but who wants to visit a bunch of sites just to keep up to date in this day and age of RSS? SC2GG’s aggregation is particularly bad: they only link to the individual commentator’s youtube pages and leave it from there.
Enter Yahoo! Pipes: a very handy web 2.0 mashup tool thing. It provides a method of aggregating and manipulating several different types of sources on the internet, but in my case, RSS feeds were available for everything, so my sources are all RSS. What I did was combine all of the youtube feeds for each SC2GG commentator, and GOMTV’s coverage, filter out non-starcraft entries, cut off entries from longer than a week ago, and sort the whole thing by the date it was published. The result: a RSS feed of Pro Starcraft games with English commentary. Or, you can view the Yahoo! Pipe itself, which allows you to get the output in several other formats, or take a look at the internals of how the pipe works, as long as you have a Yahoo! account. Pipes are pretty cool.
If anyone knows of any other sources of English commentary for Starcraft, let me know, and I’ll work on incorporating it into my feed.
Comments (0)jquery, XHTML 1.1 Strict, and NS_ERROR_INVALID_POINTER
I ran into a tiny poorly-documented problem with Firefox and jquery.
I’m developing a website using XHTML 1.1 Strict. This causes firefox to barf with a semi-helpful parsing error when the page it loads is not valid XML, instead of the usual behaviour which is to guess at what the developer meant and make and display a DOM tree anyway. This can be nice to help you validate as you code.
What it doesn’t handle well is when the changes you make to the DOM tree are not valid. I’m also using jquery in my page, and if you screw up your XML, you’ll get an almost useless error in your javascript console:
uncaught exception: [Exception... "Component returned failure code: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIDOMNSHTMLElement.innerHTML]“ nsresult: “0×80004003 (NS_ERROR_INVALID_POINTER)” location: “JS frame :: http://yourserver/include/jquery.js :: anonymous :: line 11″ data: no]
Line 0
So, viewers who are hopefully finding this page via google: if you are getting this error when modifying the DOM tree with jquery, and you’re using XHTML 1.1 Strict, then you’ve probably screwed up your XML.
Comments (1)I hate writing UI code
I really hate writing UI code. I’m bad at designing UIs in the first place, but the mindlessness of it bothers me. It is programming’s version of manual labour. There is no mental challenge in hooking up all of these signal handlers. There is no mental challenge in telling the user what part of their input was wrong. But you have to do it, and it takes forever.
When contemplating how sorry I feel for myself right now, thanks to having to work on some web programming, I came up with this simile:
Interface coding can be likened to having a big pile of rocks. You need to sort the big pile of rocks into smaller types of rocks depending on what type of rock it is. You can only carry one rock at a time. The piles are not close together.
I am not looking forward to the next few days. I haven’t even started on styling my site and am not looking forward to the nightmare of cross-browser support that comes with that.
Comments (0)Size is all that matters
Password complexity requirements really piss me off. While it isn’t my favourite activity in the world, lets do some math.
There are 52 characters in a-zA-Z. There are approximately 80 characters in a-zA-Z0-9 plus symbols. So for a password where the minimum requirements for a password include a lowercase letter, an uppercase letter, a number and a symbol, with 6 characters, is 80 to the power of 6, which is 262,144,000,000 combinations. Well gee, that’s a lot. It sure must take a bruteforcer a long time to crack that. Well, not really. This page I found for benchmarking CUDA with md5 calculates everything in millions-of-md5-hashes-per-second. That’s one computer, using consumer (albeit high-end) equipment. It wouldn’t take long to bruteforce that.
What if the password requirements were only a-zA-Z but SEVEN characters? 1,028,071,702,528 combinations. An order of magnitude more combinations, but with very lax character requirements.
Sysadmins, stop terrorizing us with your crazy password complexity requirements! If you want your passwords to be safe, make them long! A 12 to 20 character letter-only password can be far easier to memorize AND significantly stronger.
Comments (1)Nintendo DS & Homebrew
A few days ago, I purchased a Nintendo DS Lite from a friend of mine. It came with a R4DS, which much like the mod chip for the xbox, makes a decent gaming system into a great one. Aside from the nice part about being able to download all the games, a R4DS (which is only one of the many mod options) allows you to run homebrew code. That is, anyone with the desire to write and deploy their own programs to the NDS can; without the trouble of having to buy a development kit from Nintendo and pay licensing fees on the stuff we release. This isn’t to paint Nintendo in a bad light; I understand their business model and it works, but it can be frustrating sometimes when you have a cool device that is restricted by the unfortunate business policies of its corporate creator. See: iPhone, Zune.
The homebrew scene is a bit disjointed, but that’s to be expected. It could be described as both ragtag and mature. It has been possible to run homebrew code on the NDS for a few years now, so the community is well formed, and it is possible to jump in and have something running in a few days. It isn’t particularly hard to get from zero to “Hello World” in maybe a few hours of reading, downloading, and going through tutorials. After that however, the amount of available documentation starts to drop off. This isn’t the first time I’ve noticed this sort of thing either, and it seems really common with open source libraries. Ruby On Rails is a good example - tons of documentation on how to do really simple things, but as soon as you leave the basic tutorial behind, you’re on your own. That didn’t stop with RoR, and it hasn’t stopped me this time either.
The two main components to homebrew development are libnds, which is distributed as part of devkitpro, and palib.
libnds is almost essential, since it gets you off the bare metal; that is, it provides a barebones API on top of machine instructions, and little else. When I asked in the development IRC channel if there was somewhere I could find more documentation for libnds, I was told to go look at gbatek, which has the machine instruction specifications for GBA and NDS. That’s what I mean when I say ragtag. Like many tiny game development libraries, things are often functional, but not well documented, if at all.
palib is a bit higher level, providing what is actually quite a comprehensive library for all sort of things such as I/O, sprites, and filesystem support. I’ve been using it so far, but you can tell it is not a professional-level library. Still, it gets the job done, and the author has been nice enough to make a very large number of tutorials. I have no complaints yet, unless you count the sense of unprofessionalism it exudes.
The games for DS are pretty fun too. I’m currently enjoing Advance Wars: Age of Ruin, the Ace Attourney Series, Warioware, and Brain Age. I am NOT enjoying Stephanie saying her stupid Nintendog’s name over and over again, just to have it forget. I’m certain she ended up saying that stupid thing’s name over 100 times in the course of an hour.
More to come later on homebrew.
Comments (0)This Game Is Hard
Amorphous is a game that was recently featured by Kongregate. It is a fantastically simple idea, which manages to be surprisingly deep. It is very hard. I’d be interested to know if anyone I know can get the “Exterminator Badge”, which is awarded for clearing the game on “big nest” mode.
The game’s enemies remind me a lot of S.T.A.L.K.E.R.’s mutants in that you start out with fairly simple enemies like dogs and bulls, and then gradually are introduced to tougher and more-WTFy enemies. There’s a lot of fun to be had in a game that throws you some rare enemies that you’ve never encountered before, with unknown characteristics. The key to the fantastic enemy progression is that the enemies have qualitatively more difficult characteristics; the author doesn’t just give the grunt some more health and a better gun to make them harder. Eventually, you have enough experience to know how to handle the strange new characteristics. Enemies that formally meant a respawn now fall to your sword in packs.
I die a lot. The game is really hard. But the game is addictive and fantastic because as a gamer I’d like to see how creative the next challenge is.
Also, add me on Kongregate: http://www.kongregate.com/accounts/Fitzsimmons
Comments (0)The story of JobProposal
For the first half of the summer, I spent quite a bit of time working on a program for Cesmic, the company where my dad works. I used this as temporary employment while I searched for a better job.
JobProposal is the unimaginative name of some custom enterprise software that I wrote for Cesmic. Their old job and proposal tracking system was painfully old-school - it consisted of two binders, filled with a bunch of pages where employees would write in the details of the job or proposal by hand. The design itself was quite similar to that of what you would see in a relational database; a proposal could be optionally linked to a job. This was achieved by having a column in the proposal table that someone could write the id number of the job that was associated with the proposal. The system worked pretty well for a while, but scales poorly. Now that Cesmic is seeing a huge increase in work, the binder has become cumbersome and difficult to search. Software was the natural choice.
JobProposal was the first programming work I had ever done for hire, at least alone. I have done coding for hire before, but never to this scale, and I was always working for someone. Relatively speaking, JobProposal was only to be a fancy frontend to a database, like so many other internal custom software projects. From a purely creative perspective, this project is completely uninteresting. The code was not large scale, but it was far beyond trivial work as well. Just because the problem is common and has been solved already thousands of times doesn’t mean that making a friendly interface for a custom database is an easy task. UI coding is a huge pain in the ass. Also, this was the first time I would be completely responsible for every stage of the process, taking the project from conception to deployment, and maintenance.
This time, all of the choices were up to me for language, platform, and design. The desired specifications were minimal - Cesmic just needed anything that was faster than using the stupid paper binder. The experience was quite educational for me, and although the project is trivial on the grand scale, I learned a lot from the process.
One of the things I focused on for this project was the speed of coding. I picked my platform accordingly, mixing comfort from experience with easy APIs: Ruby, GTK, Activerecord, and postgres. I have a good deal of experience with all of the above. Ruby is a nice language, and I can code quickly without too many errors in it. Ruby was my top choice for languages because of its ability to have code blocks. They’re anonymous procedures that can be passed into function, much like variable arguments that contain data. This makes for streamlined UI code, since the ruby-gtk library allows you to use these blocks to specify what you want to happen when a signal occurs in the GUI. For example, here is a comparison between C++ and ruby, when I want to set up a signal that closes hides the current window when the user clicks the X:
C++:
preferencesWindow->signal_delete_event().connect(
sigc::hide(
sigc::bind_return(
mem_fun( *preferencesWindow, &Gtk::Window::hide ),
false
)
));
ruby:
win.signal_connect( "delete_event" ) { |win,event| win.hide }
My options in C++ are cumbersome. I can either write an entire new function to call a single line of code, or I can use sigc’s crazy mutation and binding functions to hack a call to the GTK hide function into the calling convention that is expected for the signal in question. I should also point out it took a lot longer than I would have liked to figure out what combinations of the mutators would result in the signature that was required. In ruby, I just put the code inside code block. Another advantage to the ruby way is that should I at some point decide that I want the delete_event signal to do two things, I can simply add code to that block. In C++, my only choice is to write a function specifically to react to that lone signal, with the two things I need done. Ruby is great for UI work for other reasons, but I think this goes far enough in illustrating my point.
I’m familiar with GTK. GTK works on the target platform. I have no complaints about GTK. No long winded explanation required.
Activerecord is a fantastic ORM, of ruby-on-rails fame. It performs well even outside of rails, although this is an unconventional setup. Making sure that I could get activerecord to perform correctly despite being out of its element was the first thing I tested. There’s nothing worse than picking the wrong tool for the job, so I wanted to find out right at the beginning. Ask me about Adobe Flex sometime, if you want to hear a story of how that can go wrong. I’m confident that my use of activerecord significantly reduced the amount of time I spent in development. It is a fantastic DB API for my purposes.
Not much to say on postgres. It is a free database, well supported, and unlike mysql, actually supports constraints. I had some previous experience maintaining postgres databases.
At last, on to the part where I feel I really learned something. When writing my own programs for hobby purposes, I would often concentrate very heavily on doing everything the Right Way(tm), and ensuring that the architecture of the program was perfect. The result of this is that I would often sacrifice code simplicity for what I envisioned as a more correct solution. Famous words from Donald Knuth:
We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.
Despite having read this quote several times, and agreeing in principle, I only recently realized that I was in many ways failing to code by these wise words. Why should I waste hours of coding time implementing a “perfect” solution to the problem when it will make a negligible difference in runtime, and needlessly add complexity to the codebase? The extra time I spend working on the improved algorithm is likely to dwarf the sum of the microscopic improvement in runtime of every execution of the program ever. However, something that I still needed to keep my mind on was scalability; while it may be acceptable for me to write a simple implementation now, the time may come in the future where the input is 1000 times larger than it is now. Only one person will be using the application right now, but what if business increases so much that some people need to work on it in parallel? These are considerations that must be taken into account when designing, and it is my belief that this kind of consideration is what separates a hastily written, or perhaps even sloppily written program from a WTF.
In some ways, my code is a little sloppy. For example, one part of the program that I’m particularly embarrassed about is that the logic of some functions are determined by the state of something in the UI; and I’m not talking about a checkbox in a configuration page. A function will do different things depending on which tab the user has selected in the main window. With unlimited time, I would have gone back to fix what I consider to be a major although not critical design flaw. This flaw does not add a significant amount of complexity to the code base, and I don’t have unlimited time. It is better that this program is deployed and working than it is that things are done Right(tm). Some could say that this type of attitude is shortsighted, and will lead to issues when maintaining the codebase. I would agree. But, in the end, I decided that the tradeoff was worth it; I can even understand why it happens so often in stereotypical terrible enterprise code.
Not everything can be perfect. Everything has some downsides. The good news is that the project was completed on time, is working, and the users are satisfied. I completed my first round of bugfixes and minor feature additions today, as part of the maintenance cycle. I leave you with this comic from Focus Shift, which I have re-hosted since I wasn’t able to find an archive on their site:
Mandatory Fun Day Sucks
The Daily WTF, for those of you who have never heard of it, is a website originally dedicated to showcasing the use of horrible source code found being used in production environments. It has since grown to doing much more; it seems to currently specialize in demonstrating horrible business practice, which I suppose is still somewhat amusing.
What isn’t amusing at ALL is the webcomic called Mandatory Fun Day, which is being featured on tdwtf. It is horrible. It is a waste of bandwidth. It is so bad that I don’t even want it showing up in my RSS feed for tdwtf anymore.
So, I went to the trouble of actually fixing this problem. My RSS reader, Liferea, has a neat feature it calls a conversion filter, supposedly for support of non-RSS feeds, with the help of a script. It passes the data it fetches from the source URL to a script’s STDIN, and expects standard RSS from the script’s STDIO. I fashioned a script to get rid of the scourge that is Mandatory Fun Day. Here it is:
#!/usr/bin/env ruby require 'rexml/document' doc = REXML::Document.new( $stdin ) #select and delete all of the items with "Mandatory Fun Day" as the category doc.root.elements.delete_all( "/rss/channel/item[category/text()='Mandatory Fun Day']" ) doc.write
Slap that in a file, set it executable, and re-add the tdwtf feed. When adding, hit advanced, and enable the conversion filter. Point it at the script. Voila, no more Mandatory Fun Day. And people say XML sucks.
Comments (3)