Narrowing The Choices
Context
Andy Matuschak recently made a blog post regarding first thoughts on a package manager for OS X. One of the key ideas of his post was that we could integrate it into Spotlight, allowing for easy use by non-technical users. In my entry, What Keeps Me Awake I detailed my own reasoning that led me to a similar conclusion about OS X needing a package manager and building it into Spotlight.
The comments on Andy’s blog have started the discussion about what kind of thinking we need to do before we even go near implementing this. One point, brought up by Leonardo Cassarani, is that we’re going to need a pretty good algorithm to determine what relevant apps should be displayed to the user. This post is about some of the things I think we should take into consideration for this algorithm.
First I’ll cover a couple of usage scenarios to give a better idea how I think the app should work, in the process demonstrating an idea that I think might help with the relevancy algorithm problem. Then I’ll swerve into a discussion of why I think this solution might be the one. After that I’ll address a concern I saw expressed by Leonardo regarding how the relevancy algorithm could be skewed.
Also, for the first time on this blog, comments are open. How do you folks think we should determine which apps make the search results?
Usage Scenarios
Let’s try a couple scenarios to see how we expect the software to illustrate how I have the software working in my head. I’ll make an example of my Mom here, since she’s a pretty typical end user. She’s comfortable with using what she uses everyday, but exploring new territory isn’t one of her strengths.
- My Mom calls me and says she has to ‘upload’ a few ‘files’ to an ‘FTP site’. I’d normally tell her to Panic and get Transmit, since that’s what I use so I’ll be able to walk her through the rest. I tell her to go to the little magnifying glass in the top right corner and click it. Type in ‘Transmit’. She sees something like this as a spotlight result:
Suggested Applications Transmit – Next Generation FTP Client
15 Day Free Demo – $29.95Very carefully she clicks on the “15 Day Free Demo” text, because she’s sure she’ll get a bill in the mail if she clicks on $29.95. The Transmit icon, maybe with an indeterminate progress indicator superimposed on it, bounces in the dock and a small window comes up showing how long until Transmit will launch. It says something like “The free trial of Transmit is being downloaded to your computer. It will launch as soon as it’s finished, in about 2 minutes” with a progress bar. Keep in mind that she’s on the lowest tier of ADSL she can get, since it’s all ‘high speed’ and she thinks her computer is being slow, so we need that progress bar to let her know something is happening.
When it launches, she uses it for a few minutes to ‘upload’ those ‘files’ and quits it. When she quits it a little dialog pops up with something like “Thank you for trying Transmit. Would you like to keep a copy on your computer for the next time you need to use it?” Whatever she clicks, the package manager sends the search term “Transmit” and whether she kept a copy or not to some central server. That data is going to come into play in the next scenario.
Before I get off the phone I tell my Mom that the next time she needs to do something and she doesn’t have a program for it, type what she needs to do into the little magnifying glass and it’ll show her some tools that will do it. She writes down this instruction down in the little notebook she uses to keep all the things I tell her about how her computer works.
- A month later she sits down at her computer. She collects silver teapots as a hobby, and has decided that it’s time to put a few on eBay. She’s not actually sure how she’s supposed to do that, but she looks at things on eBay on the computer, so she must use it to sell things on eBay too. She remembers the instruction to try to use the magnifying glass to find programs to do things.
She types in “eBay” and sees a result like this:
Suggested Applications GarageSale – Slick eBay Client
3 Item Demo – $29.99eBay Watcher – eBay Item Monitoring Widget
Limited Demo – $10.00With all the same careful actions as before, she patiently waits for GarageSale to download and run. When she sees the first configuration screen, she decides that maybe she’ll just wait until my next visit and have me show her how it works then. When she quits the app, she clicks “No” when it asks if she wants to keep a copy.
On the centralized server, some weighting system determines that GarageSale is ever so slightly less likely to be what people searching for ‘eBay’ are looking for. (I should note that I’ve never actually used GarageSale, and I’m sure it’s a fine application.)
Narrowing The Choices
It’s likely a sane choice to say that if there’s an exact match by application name, we should return that application as the first suggested app. Note that we should still return more than one app though, otherwise the system is too easy to game by naming your RSS reader “Blogs”.
Past that obvious thought, it becomes a little harder. How do we figure out what apps are the best search results? Download stats don’t work because you might try several apps before finding one that’s actually what you wanted. iUseThis or VersionTracker ratings are skewed towards a particular demographic (power users) which isn’t representative of the overall OS X demographic. Editorial decisions don’t work, since you’d have to have a whole legion of editors doing nothing but trying to anticipate search terms and evaluating apps.
In my usage scenarios, you noticed the dialog which says “Thank you for trying AppName. Would you like to keep a copy on your computer for the next time you need to use it?” At first glance, this is a bad idea, since it’s an extra click for the user and breaks their flow. Without something like it though, we’re probably looking at a lot of cluttered ‘Applications’ directories as people try three or four apps for a minute and only end up using one.
Even more important than the cleanup issue is the relevancy issue. Each ‘keep’ or ‘don’t keep’ click is effectively a ‘This is what I wanted’ or not vote. The question of how to get initial values for this system is still a problem, but with this idea we end up having the relevancy rankings adjust themselves over time to become more accurate.
Concerns About Skewed Rankings
Leonardo brought up the concern that free applications which offer a worse user experience are promoted over better applications which cost. I’m not sure this is actually a concern. Isn’t cost part of the user experience? If there are two applications which do the same thing, but the paid one is more polished, shouldn’t the polish have to be enough to offset the cost to the user?
In addition to that point, I don’t think we’d see a large skewing towards free apps if we adopt some variant of the ranking system I’ve just proposed. Because the ranking is based on what the users impression of their first experience is, sane shareware trials will keep paid and free apps on equal footing, at least as far as first impressions go.
Closing Thoughts
There’s still plenty of thought to put into this thing before the first line of useable code is going to be typed. How do we deal with more ambiguous terms? How do we get an initial weighting for these terms? I think this will all get resolved with time, but I’m of the mindset that there’ll be a lot of words about this package manager before it actually gets built.
How do you think an apps relevancy should be determined?
December 22nd, 2007 at 12:44 pm
Good thoughts, Bob.
I like the idea of the “would you like to keep this app around?” dialog, but consider that if this system becomes popular, those exit-interrupting modal dialogs would be pretty frequent. If you click No, does it ask again next time you use it? Seems like the best answer would be no, but then we need some other interface for keeping copies on the drive.
If not for the voting system issue, I was thinking a system that weighs whether applications should be kept around by time since last launch, adjusted by how much time the user has spent in the app. For instance, if a user has used one app quite a lot, he can avoid using it for a month or two, and it won’t be erased. With a simple slider in the prefs for how lnog to keep things around.
As I was telling Leonardo, I think the best way to do rankings on this is just to see how long, in total, the user has the app open. But that’s creepy. Of course, if I had my way in creepiness, I’d go by how many events were sent to to the app: that’d fix rankings for always-open apps and for people who forget to close things.
I don’t really want this to be a keyword kind of search. That encourages developers to come up with a huge list of words, which is both a hassle and not very useful. I’m thinking something a little fuzzier, a little Google-like. What do pages talking about that app have on them? Let’s pick out statistically improbably words and weight them by frequency. Something like this.
December 22nd, 2007 at 11:19 pm
Andy -
Good point about what happens next time you use it. I hadn’t considered that some users will probably choose to redownload an app every time they use it. Ideally there would be four choices: “Keep”, “Don’t Keep”, “Keep Everything”, “Don’t Keep Unless I Hit Download”. Obviously that’s far too many choices.
When initially thinking about this idea, it struck me that the only place in the system you’re currently prompted to keep an app is if you add a widget to the dashboard. In that case it surrounds the app with a small shaded area with a thick border around it. You get two options, “Delete” and “Keep”. We could probably do a similar thing with apps. At any point while running it, they could press “Keep” and the app would be stored locally. If they pressed “Delete”, a little dialog along the lines of “AppName will be deleted from your computer when you quit the application.” pops up and then fades away a moment later. If they quit without keeping the app, it’s counted as a delete. The next time they download the app, the ‘Delete or Keep’ border isn’t shown, unless they quit last time without making a choice.
This has the same effect as my ‘quitting dialog’ proposal above, but without the need for a modal dialog. It also uses the same procedure as Dashboard, resulting in further consistency with the existing system.
In your initial mockup you had three buttons which looked to be “Info”, “Run” and “Download and Run”. If they’ve hit “Delete” previously, I think hitting the “Download and Run” button should reverse their delete vote.
I’m against the idea of apps that delete after a certain period of inactivity, depending upon how frequently the app is used. The idea of a prefs slider to determine how long it’s kept might seem simple to CompSci people, but I can’t imagine explaining this to an end user.
“Yes, once you run it it keeps a copy on your system, unless you don’t use it for a while.”
“Oh…how often do I have to use it to keep it?”
“That really depends on how often you use it.”
I agree that ideally you’d be able to measure how much a user actually uses an app, as opposed to just if they liked it or not, but if we want to get this eventually adopted by Apple that’s out of the question. Remember how big of a press nightmare they had when the iTunes ministore started sending back data so it could display similar albums? And they weren’t even keeping that data.
I also agree that it shouldn’t be a keyword search. I didn’t mean to imply that the developer should be the one to supply the associated terms. I’m starting to think of using some sort of weighted graph model to determine the most relevant choices, but I haven’t even thought about how it’d go about working. Using keywords based on review/product page/etc crawling and then further weighting them with user feedback in the form of keep/delete might be the way to go.
December 24th, 2007 at 1:50 am
You’re suggesting we surround the whole app the entire time the user’s using it with a Dashboard-like border? I mean, I’m all for familiar metaphor, but that’s probably a bit much. I think that’ll lead to people just clicking “Keep” to make the bezel go away so they can try out the app unencumbered.
We could potentially make the bezel appear when the user asks to *quit* the app. But that’s still probably less than ideal. Hrmmmmm.
I think you’re probably right about the machine learning. I asked my mom, and she said she’d just prefer to be asked.
December 26th, 2007 at 7:00 am
Hi again,
in my opinion we’re dealing with three distinct issues here. First: How do we determine what the user wants? Second: How are apps classified, presented, ordered? Third: How do we install them, keep them etc.
I’ll just write down my thoughts on this, and you can bash me afterwards.
For the first issue we have a tradeoff between creepiness and effectivity. The creepy way: What the user wants is determined by what the user is. What the user is, can be approximated, for our case, by what the user uses. Or, simply put: what he already downloaded using iDontKnowHowToNameIt, what he uses regularly (event count is a good idea). We can mine on this and find that people that use some blog frontend will likely want Scivener when looking for writing. Bad example, but you get the drift. This is, of course, horribly creepy. But we could keep the usage info and data locally and send the queries as anonymously as the web permits. No solution, I know, but it all converges to a Google-knows-you world, then we can just stick with it, reap the advantages for the user and keep him away from the bad stuff as far as possible.
The non-creepy way will be voting, but this is very community-dependent. Maybe if the user classifies himself (or we could access the what-do-you-use-your-Mac-for info everyone has to provide upon install), we’d have a slight clustering that might already prove helpful. Anyhow all of this is optional. If the user says no, he’ll get the mediocre results and keeps his privacy.
As for information and classification of the apps in terms of functionality, I am in favor of a Yahoo-ish manual catalogue solution. Text like ‘the next-gen ftp client’ as provided by the developer is marketing speak in most cases and thus probably useless. A Wikipedia-like software archive where the programs can be described, rated and catalogued by a community (and, of course, the developers) might provide fairer and more useful information. It’s a bit of a long shot from now though.
Concerning the installation and keeping we have to realize the following: This will end in web-apps. I think we have to think in these terms as well. The interesting question will not be about keeping or shunning, because broadband will be everywhere and any given app won’t take much longer than a few minutes to load (unless I largely overestimate your broadband over in the states
). The real question the user will have to answer for himself (and to us) is: Do I need this app offline, when I’m on the train, at granny’s, in the park?
I suggest the following: We ask the user for a certain amount of disk space to be used as app cache. This is where we keep downloaded apps around. They get updated if the user starts the app and there’s a more recent version online. They get deleted if we’re out of space, there’s new apps pushing in and the app has fallen below some usage threshold. Metadata and settings are, of course kept around. If there’s no app below the threshold, we can ask for more disk space.
We can ask about the offline keeping after the first app quitting (your bezel idea, Andy). Apps to be kept offline go into Applications and do not count towards our cache space.
There’s another thing to think about: The dev side and the dependencies. As other packet managers do, we’ll need to enforce a strict dependency write-down and strict versioning to ensure auto-updating app X to framework Z doesn’t break anything. Notifications to devs about new versions of pakets their software depends on as well as an auto-marker for abandoned software (as in critical bug/breach in dependency, but no update from dev) would be of use. Maybe we could include a Sparkle-like in-app Framework that’ll allow for a menu item concerning the caching, keeping rating etc. Devs would be free to use it and it might save fumbling around on some PrefPanes. Although it also means more clutter.
Ok. I clearly lost my way now. This needs a wiki badly.
Sorry for the messy thoughts,
Juri
December 27th, 2007 at 4:13 am
Hello -
I’ve been silent for several days, away from my network connection. While I’ve been gone, I’ve been thinking a bit more about this problem. I think Juri has stated it quite well; we’re looking at three distinct issues here.
1 – How do we figure out what the user wants. If they look for ‘writing’, do they want blog software or screenplay software or a word processor? This is the algorithm that Leonardo spoke about. The closest parallel problem I can think of is “what contextual ads do we show to search engine users when they search for a term?”
Unfortunately, the creepier the algorithm, the more likely that we’re going to be getting the right result. Things like your browser cache, your previous app download patterns, your already installed apps, etc. could all be used to get the user the most relevant results. I think Juri might have hit it on the head with the idea of keeping info locally. I’d like to take it a step further.
Suppose we made the web side of the package manager extremely verbose. Results would be returned in order of download popularity, then the user-side software would perform most of the grunt work to determine which results to display. Writing might return an extreme number of results, but the user could receive reasonably targeted results without any of their personal data being exposed to the server.
That being put forth, we still need to figure out that algorithm.
2 – How are apps classified/presented/ordered? I’m all for some kind of wiki that we could mine for the most useful terms as a way to refine results in the long term, but to get this kicked off we need something short term.
I think we’re all agreed that we can’t rely upon developers as the sole source of terms. Marketing speak is obviously a concern, as is the issue that developers would have to actively participate. A manual cataloging system might be ideal, but that would require quite a bit of manpower.
My vote goes for Andy’s idea about harvesting relevant terms from webpages. That’s a whole algorithm in itself of course.
3 – How do we install them and when do we keep them permanently?
I’ve been rethinking the dashboard style bezel idea. It hadn’t struck me that the user might see the app as somehow disabled before choosing an option. If that’s the case, it’s probably best the throw out the idea altogether. Popping up the bezel on quit would probably be confusion as well, and just as modal as the dialog it was intended to replace.
Forgive my wandering for a moment while I try to piece together a thought:
Juri said “…we have to realize the following: This will end in web-apps”. I think he’s right. What we’re talking about is effectively building a way to give desktop apps all the advantages that web apps currently are gaining ground with, but without giving up the benefits of desktop apps.
I initially proposed the ‘keep’ button as a way for users to vote-without-voting for the apps they liked. If we compute the contextual results algorithm locally, suddenly we have a lot more data to work with than a right/wrong vote. We aren’t nearly as dependent on that vote.
Along the way, the keep button also became the way for a user to place an application into /Applications. Andy’s original mockup had a download button right in the spotlight results. If we’re no longer reliant upon the keep vote to determine an apps place in the results, then we can use that download button as the way to place an app into /Applications and simply cache other apps.
I’m against anything where we have to ask the user for disk space, since that’s going to drive off a lot of the people this is going to help. Perhaps a reasonable default like 500MB or something and a preference pane with a ‘cache size’ slider and ‘empty app cache’ button? As long as this is a third party install it needs to have some place to put an uninstall button anyhow.
So we install them automatically in a cached space (user data isn’t put in this place obviously). Every time they’re used they’re bumped to the end of the delete queue. If more disk space is needed, the front of the delete queue is remove. We keep an app permanently in /Applications (not counted in the cache allocation) when the ‘download’ button is pressed. A sparkle-esque developer plugin would be available for developers. If the app is in the cache location, it would present a menu item to ‘Keep AppName In My Applications Folder” or something similar. No keep button, no modal dialog. If you happen to be on an airplane, only your cached apps are considered under ’suggested applications’.
A mailing list might help keep this all a bit more organized.
December 29th, 2007 at 3:06 am
Sorry that I’ve been silent recently: the holidays have been attacking.
I like Juri’s compromises of caching some apps to some configurable max size (with a reasonable default), though I’m still not pleased with any of the mechanisms for determining if an app should be kept yet.
Another idea I had:
There are a number of paths by which a user could come across an app he wants to try. Looking for something in particular is only one. Since I already have all the applications I need on a day-to-day basis, I only ever install new apps when I’m surfing the net and see that something new has come out to solve an existing problem. This is a web-based experience, but there’s still that pesky download link. It’s be nice to have some kind of runapp:// URL handler or something so that a “try MyApp!” button on the site could make it happen. Security’s an issue, of course, but it’s something to think about.
December 29th, 2007 at 3:08 am
Additional note: several people have suggested that we create a framework to enable developers to choose to add menus to control per-app settings for this system.
There’s no need: our daemon can just add the menu in question to every app’s main menu. Not hard.