<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Vedant Misra</title>
	<atom:link href="http://vedantmisra.com/feed/" rel="self" type="application/rss+xml" />
	<link>http://vedantmisra.com</link>
	<description>Data-driven cogitation. Not too much prevarication.</description>
	<lastBuildDate>Sun, 05 Feb 2012 20:39:11 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.1</generator>
		<item>
		<title>Markov chain paper title generator</title>
		<link>http://vedantmisra.com/2012/01/markov-chain-paper-title-generator/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=markov-chain-paper-title-generator</link>
		<comments>http://vedantmisra.com/2012/01/markov-chain-paper-title-generator/#comments</comments>
		<pubDate>Thu, 05 Jan 2012 23:41:07 +0000</pubDate>
		<dc:creator>Vedant</dc:creator>
				<category><![CDATA[Article]]></category>
		<category><![CDATA[algorithms]]></category>
		<category><![CDATA[computers]]></category>
		<category><![CDATA[data]]></category>
		<category><![CDATA[science]]></category>

		<guid isPermaLink="false">http://vedantmisra.com/?p=395</guid>
		<description><![CDATA[Gene Stanley is a prolific and influential physicist who has been one of the biggest pioneers of interdisciplinary science of the last several decades. His H-Index is an obscene 111. I work on econophysics, which he basically invented, so I encounter his papers regularly. After seeing a few dozen of his econophysics papers, their titles [...]]]></description>
			<content:encoded><![CDATA[<p>Gene Stanley is a prolific and influential physicist who has been one of the biggest pioneers of interdisciplinary science of the last several decades. His <a href="http://en.wikipedia.org/wiki/H-index">H-Index</a> is an obscene 111. </p>
<p>I work on <a href="http://polymer.bu.edu/~hes/econophysics/">econophysics</a>, which he basically invented, so I encounter his papers regularly. After seeing a few dozen of his econophysics papers, their titles start to blur together. This is no surprise&#8212;he has published so much in econophysics that many of the same terms appear in his paper titles.</p>
<p>So I wrote a Markov title generator and trained it on Gene Stanley&#8217;s econophysics papers to see if it could generate Stanleyesque paper titles. See for yourself:</p>
<blockquote><p>
The Distribution of Commodity Price Fluctuations.<br />
Correlation in Complex Organizations.<br />
Scaling Behavior in the Random Matrix Theory Approach to the Indian Stock Index.<br />
Statistical Regularities in Stock Prices.
</p></blockquote>
<p>I would totally read these papers. Some of them even sound pretty interesting. Sadly, they don&#8217;t actually exist.</p>
<p>Try generating your own paper titles with <a href="https://github.com/vedant/markov-chain-generator">this python library</a> I wrote. It generates chains of arbitrary length and is optimized for clarity and not speed, and it&#8217;s based on <a href="http://agiliq.com/blog/2009/06/generating-pseudo-random-text-with-markov-chains-u/">shabda&#8217;s on agiliq.com</a>.</p>
<p>When called from the command line, it&#8217;ll scrape titles from Google Scholar if you feed it an author name as a command line argument. So</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;">python markov.<span style="color: black;">py</span> <span style="color: #483d8b;">&quot;HE Stanley&quot;</span></pre></div></div>

<p>will scrape 300 of Stanley&#8217;s paper titles as the source corpus. This works best when the author name is provided as first and middle initials and a last name, as shown.  Caution: scraping Google scholar violates their TOS.</p>
<p>If you&#8217;re using a small corpus and long Markov chains, you&#8217;ll end up with lots of actual strings from the corpus, and no fake ones.  If this happens, experiment with the second parameter to the constructor for the class &#8220;MarkovGenerator.&#8221;</p>
<p>See also <a href="http://pdos.csail.mit.edu/scigen/">SCIgen</a>, which generates entire papers, but cannot be seeded with a particular author or category.</p>
<p>Code on <a href="https://github.com/vedant/markov-chain-generator">github</a>.</p>
<p>See comments on <a href="http://news.ycombinator.com/item?id=3431242">HackerNews</a>.</p>
]]></content:encoded>
			<wfw:commentRss>http://vedantmisra.com/2012/01/markov-chain-paper-title-generator/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Elevator algorithms</title>
		<link>http://vedantmisra.com/2011/12/elevator-algorithms/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=elevator-algorithms</link>
		<comments>http://vedantmisra.com/2011/12/elevator-algorithms/#comments</comments>
		<pubDate>Sat, 24 Dec 2011 18:34:27 +0000</pubDate>
		<dc:creator>Vedant</dc:creator>
				<category><![CDATA[Article]]></category>
		<category><![CDATA[algorithms]]></category>
		<category><![CDATA[computers]]></category>
		<category><![CDATA[engineering]]></category>
		<category><![CDATA[society]]></category>

		<guid isPermaLink="false">http://vedantmisra.com/?p=413</guid>
		<description><![CDATA[A recent post on HN got me thinking (and reading) about elevator scheduling algorithms.  As it turns out, this is an active area of extensive research. Manufacturers tend to use slightly different algorithms and treat them as trade secrets. But in practice, their algorithms are similar, because the theoretical optimization criteria are roughly the same: provide [...]]]></description>
			<content:encoded><![CDATA[<p><a href="https://news.ycombinator.com/item?id=3351649">A recent post on HN</a> got me thinking (and reading) about elevator scheduling algorithms.  As it turns out, this is an active area of extensive research.</p>
<p>Manufacturers tend to use slightly different algorithms and treat them as trade secrets. But in practice, their algorithms are similar, because the theoretical optimization criteria are roughly the same:</p>
<ul>
<li>provide even service to each floor</li>
<li>minimize how long passengers wait for an elevator to arrive</li>
<li>minimize how long passengers spend to get to their destination floor</li>
<li>serve as many passengers as possible</li>
</ul>
<p>The single elevator case is not very interesting, nor is the case when the passenger can&#8217;t specify his direction of travel when making the call (one button per floor), so I&#8217;ll instead discuss only the case where there are multiple elevators, and two buttons on each floor.</p>
<p>There are many criteria to consider in elevator scheduling. For example, people have predictable behavioral patterns that must be addressed, including the uppeak and downpeak&#8212;respectively 9AM and 5PM, in many office buildings&#8212;which are when elevator efficiency matters the most. There is often a 1-2 hour two-way peak (lunchtime) to address as well. Algorithms should consider whether an elevator is full before assigning it to an elevator call. Sometimes, some blocks of floors have predictably higher interblock or intrablock traffic than other blocks. Often, calls on some floors (executive floors, for example) are given higher priority than others (basements). All of these factors increase the algorithm sophistication.</p>
<p>Still, they tend to be based on the four classic group traffic control algorithms.</p>
<ul>
<li><strong>Nearest Car (NC): </strong>Elevator calls are assigned to the elevator best placed to answer that call according to three criteria that are used to compute a figure of suitability (FS) for each elevator. (1) If an elevator is moving towards a call, and the call is in the same direction, FS = (N + 2) &#8211; d, where N is one less than the number of floors in the building, and d is the distance in floors between the elevator and the passenger call. (2) If the elevator is moving towards the call, but the call is in the opposite direction, FS = (N + 1) &#8211; d. (3) If the elevator is moving away from the point of call, FS = 1. The elevator with the highest FS for each call is sent to answer it. The search for the &#8220;nearest car&#8221; is performed continuously until each call is serviced.</li>
<li><strong>Fixed Sectoring Common Sector System (FSO): </strong>The building is divided into as many sectors as there are elevators. Elevators in each sector prefer calls in that sector.</li>
<li><strong>Fixed Sectoring Priority Timed System (FS4): </strong>The building is divided into up sectors and down sectors, and elevators only ever treat down calls in down sectors and up calls in up sectors. Each sector has a priority level, which increases the longer the passengers wait. The rate of increase can vary from sector to sector and over time.</li>
<li><strong>Dynamic Sectoring System (DS): </strong>Floors are grouped into dynamic sectors. Each elevator is allocated to a sector in the sector definition, and the sectors change size and location based on the position of moving and idle elevators.</li>
</ul>
<p>Modern control systems do even more than this. Some of them dynamically compute cost functions for passengers waiting on an elevator. Stochastic traffic control systems empirically compute the distribution of response times and try to make it as Gaussian as possible (wait times should be consistent; there shouldn&#8217;t be some times when elevators respond instantly and others where they take a while). Some advanced techniques use fuzzy logic schedulers (Ho and Robertson 1994), genetic algorithms (Siikonen 2001, Miravete 1999), and neural networks (Barney and Imrak 2001).</p>
<p>Most of this information is paraphrased from UK-based lift consultant Gina Barney&#8217;s book &#8220;Elevator Traffic Handbook: Theory and Practice.&#8221; A most uplifting read.</p>
<p>This post originated as <a href="http://www.quora.com/Is-there-any-public-elevator-scheduling-algorithm-standard#">an answer on Quora</a>.</p>
<p>See discussion on <a href="http://www.reddit.com/r/programming/comments/npevh/actual_elevator_algorithms/">reddit</a>, <a href="http://news.ycombinator.com/item?id=3389374">hackernews</a>, and <a href="https://twitter.com/#!/search/http%3A%2F%2Fvedantmisra.com%2F2011%2F12%2Felevator-algorithms%2F">twitter</a>.</p>
]]></content:encoded>
			<wfw:commentRss>http://vedantmisra.com/2011/12/elevator-algorithms/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Rooting Android</title>
		<link>http://vedantmisra.com/2011/12/rooting-android/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=rooting-android</link>
		<comments>http://vedantmisra.com/2011/12/rooting-android/#comments</comments>
		<pubDate>Wed, 21 Dec 2011 12:00:59 +0000</pubDate>
		<dc:creator>Vedant</dc:creator>
				<category><![CDATA[Article]]></category>
		<category><![CDATA[computers]]></category>
		<category><![CDATA[security]]></category>

		<guid isPermaLink="false">http://vedantmisra.com/?p=417</guid>
		<description><![CDATA[Android was rooted quite a while ago by a hacker who posts at c-skills.blogspot.com.  Like many others, I tried out the exploit he installed to see if I could make my phone more interesting (this is something it&#8217;s now okay to admit to).  This got me wondering about whether the exploit for Android is generalizable&#8212;is there [...]]]></description>
			<content:encoded><![CDATA[<p>Android was <a href="http://en.wikipedia.org/wiki/Rooting_(Android_OS)">rooted</a> quite a while ago by a hacker who posts at <a href="http://c-skills.blogspot.com/">c-skills.blogspot.com</a>.  Like many others, I tried out the exploit he installed to see if I could make my phone more interesting (this is something <a href="http://www.copyright.gov/1201/">it&#8217;s now okay to admit to</a>).  This got me wondering about whether the exploit for Android is generalizable&#8212;is there any way it can be applied to Linux generally?</p>
<p>Short answer: no.</p>
<p>Rage Against the Cage (RAtC) exploits the fact that the Android Debug Bridge daemon (adb) on Android devices starts as root by default, and calls setuid to drop its privileges to those of a shell account.  The ADB daemon is what runs on Android phones to enable Android software developers to communicate with the phones they&#8217;re testing their software on.  This kind of issue doesn&#8217;t exist in Linux in general, so the exploit probably can&#8217;t be generalized.  Nonetheless, here is some more detail on how RAtC works.</p>
<p>In Linux, a system-wide resource limit, RLIMIT_NPROC, defines the maximum number of simultaneous processes allowed by the system (Android, in this case). RAtC checks this limit, and spawns processes that do nothing until Android&#8217;s limit is reached. This is called a fork bomb.</p>
<p>With the number of processes maxed out, RAtC kills the running adb process on the phone. The process that starts adb when the phone is turned on tries to ensure that adb is always running, so it restarts adb.  There&#8217;s a race condition for that parent process to overcome here, because the adb process must be spawned while RLIMIT_NPROC is already nearly maxed out.</p>
<p>Adb restarts by default as root, which it needs to do so that developers can properly debug their software. It then quickly checks if it does indeed need to keep running as root, or if it can downgrade its privileges because root access isn&#8217;t necessary. The former is only true when Android is running in a sandboxed emulator on a developer&#8217;s computer or if the Android build is a debugging version, so it tries to drop its privileges using setuid.</p>
<p>But RAtC&#8217;s fork bomb has already maxed out the number of processes already running, so the call to setuid fails, and adb keeps running as root. With adb running as root, we can do whatever we want on the device.</p>
<p>This is also <a href="http://www.quora.com/What-are-the-exact-mechanisms-flaws-exploited-by-the-rage-against-the-cage-and-z4root-Android-exploits#ans883688">a post on Quora</a>.</p>
]]></content:encoded>
			<wfw:commentRss>http://vedantmisra.com/2011/12/rooting-android/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Market manipulation in the financial crisis</title>
		<link>http://vedantmisra.com/2011/12/market-manipulation-in-the-financial-crisis/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=market-manipulation-in-the-financial-crisis</link>
		<comments>http://vedantmisra.com/2011/12/market-manipulation-in-the-financial-crisis/#comments</comments>
		<pubDate>Tue, 20 Dec 2011 19:20:08 +0000</pubDate>
		<dc:creator>Vedant</dc:creator>
				<category><![CDATA[Research]]></category>

		<guid isPermaLink="false">http://vedantmisra.com/?p=409</guid>
		<description><![CDATA[I authored some work recently in which we identify an instance of potential market manipulation. Here&#8217;s the page on NECSI&#8217;s website. Here are reddit posts about it in economics, politics, math, finance, and occupywallstreet.  Here&#8217;s an HN post about it. It was covered by BoingBoing, PhysOrg, ScienceNews, Technology Review, ZeroHedge, Naked Capitalism, and others.]]></description>
			<content:encoded><![CDATA[<p>I authored some work recently in which we identify an instance of potential market manipulation.</p>
<p>Here&#8217;s the page on <a href="http://necsi.edu/research/economics/bearraid.html">NECSI&#8217;s website</a>.</p>
<p>Here are reddit posts about it in <a href="http://www.reddit.com/r/Economics/comments/nhvq6/evidence_of_market_manipulation_in_the_financial/">economics</a>, <a href="http://www.reddit.com/r/PoliticsPDFs/comments/niemn/evidence_of_market_manipulation_in_the_financial/">politics</a>, <a href="http://www.reddit.com/r/math/comments/ngnnj/evidence_of_market_manipulation_in_the_financial/">math</a>, <a href="http://www.reddit.com/r/finance/comments/nicqk/evidence_of_market_manipulation_in_the_financial/">finance</a>, and <a href="http://www.reddit.com/r/occupywallstreet/comments/ng4pn/evidence_of_market_manipulation_in_the_financial/">occupywallstreet</a>.  Here&#8217;s <a href="http://news.ycombinator.com/item?id=3380626">an HN post</a> about it.</p>
<p>It was covered by <a href="http://boingboing.net/2011/12/21/complex-systems-institute-clai.html">BoingBoing</a>, <a href="http://www.physorg.com/news/2011-12-mystery-predators-contributed-fiscal-collapse.html">PhysOrg</a>, <a href="http://www.sciencenews.org/view/generic/id/336990/title/Smells_like_a_bear_raid">ScienceNews</a>, <a href="http://www.technologyreview.com/blog/arxiv/27422/">Technology Review</a>, <a href="http://www.zerohedge.com/contributed/market-manipulation-financial-crisis">ZeroHedge</a>, <a href="http://www.nakedcapitalism.com/2011/12/links-121811.html">Naked Capitalism</a>, and others.</p>
]]></content:encoded>
			<wfw:commentRss>http://vedantmisra.com/2011/12/market-manipulation-in-the-financial-crisis/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Quantifying trendy names</title>
		<link>http://vedantmisra.com/2011/10/quantifying-trendy-names/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=quantifying-trendy-names</link>
		<comments>http://vedantmisra.com/2011/10/quantifying-trendy-names/#comments</comments>
		<pubDate>Sun, 02 Oct 2011 18:00:36 +0000</pubDate>
		<dc:creator>Vedant</dc:creator>
				<category><![CDATA[Article]]></category>
		<category><![CDATA[data]]></category>
		<category><![CDATA[science]]></category>
		<category><![CDATA[society]]></category>

		<guid isPermaLink="false">http://vedantmisra.com/?p=273</guid>
		<description><![CDATA[A friend recently remarked that he hated &#8220;trendy&#8221; names like &#8220;Brody&#8221; and &#8220;Jayden.&#8221; Yahoo answers suggests not only that he has company [1][2][3], but that there are certain patterns common to trendy names that mark them as neo-praenomenisms, like the replacement of &#8220;a&#8221; with &#8220;ay&#8221; or &#8220;ae&#8221; to represent the sound /eI/ (s/Jaden/Jayden), and the [...]]]></description>
			<content:encoded><![CDATA[<p>A friend recently remarked that he hated &#8220;trendy&#8221; names like &#8220;Brody&#8221; and &#8220;Jayden.&#8221; Yahoo answers suggests not only that he has company [<a href="http://answers.yahoo.com/question/index?qid=20110718234122AAm1oKt">1</a>][<a href="http://answers.yahoo.com/question/index?qid=20110816024709AAP4WI8">2</a>][<a href="http://answers.yahoo.com/question/index?qid=20110727093017AAKkb53">3</a>], but that there are certain patterns common to trendy names that mark them as neo-<a href="http://en.wikipedia.org/wiki/Praenomen">praenomenisms</a>, like the replacement of &#8220;a&#8221; with &#8220;ay&#8221; or &#8220;ae&#8221; to represent the sound <a href="http://en.wikipedia.org/wiki/International_Phonetic_Alphabet_chart_for_English_dialects">/eI/</a> (s/Jaden/Jayden), and the replacement of &#8220;i&#8221; with &#8220;y&#8221; to represent <a href="http://en.wikipedia.org/wiki/International_Phonetic_Alphabet_chart_for_English_dialects">/I/</a> (s/Kaitlin/Kaitlyn).</p>
<p>One hypothesis to explain why &#8220;trendy&#8221; names attract revulsion is that one is likely to have experienced a greater degree of immaturity with younger people, and people with trendy names might be of lower average age than people with more traditional names. That is, trendy names might map more strongly to a younger part of the age spectrum. Wolfram Alpha confirms that people with trendy names are younger, on average:<br />
<a href="http://www.wolframalpha.com/input/?i=brody"><img class="aligncenter size-full wp-image-333" title="ssa-names_wolfram-alpha_brody" src="http://vedantmisra.com/wp-content/uploads/2011/10/ssa-names_wolfram-alpha_brody.gif" alt="" width="496" height="195" /></a><br />
<span id="more-273"></span><br />
I wanted to quantify that intuition by generating a set of names that is &#8220;trendy&#8221; by some statistical measure using the <a href="http://www.ssa.gov/oact/babynames/">Social Security Administration&#8217;s names database</a>, which contains time series of the number of children born in the US with each given name. I picked an arbitrary trendy-sounding name&#8212;&#8221;Brody&#8221;&#8212;and identified names with which its time series had the highest correlation. Below are the time series, normalized by max value and smoothed in a three year window, for the 50 names most highly correlated with &#8220;Brody&#8221;. The darkest line is &#8220;Brody,&#8221; with lighter lines subsequently less correlated.<img class="aligncenter size-full wp-image-276" title="ssa-names_highest-correlates_brody-M" src="http://vedantmisra.com/wp-content/uploads/2011/10/ssa-names_highest-correlates_brody-M.png" alt="" width="600" height="450" /><br />
What are the least correlated names? Are they Biblical names? Are they names that sound &#8220;classical?&#8221; Here are their time series, also normalized by max value and smoothed in a three year window:<br />
<img class="aligncenter size-full wp-image-276" title="ssa-names_highest-and-lowest-correlates_brody-M" src="http://vedantmisra.com/wp-content/uploads/2011/10/ssa-names_highest-and-lowest-correlates_brody-M.png" alt="" width="600" height="450" /><br />
These names seem to experience a surge of interest in the late 1800s, followed by a dip in the early 20th century; then they peak out around 1920, and wane steadily in popularity to the present day, to the point that they are all quite unpopular. Here are the names themselves:</p>
<table class="data-table">
<tbody>
<tr>
<td style="border: 0px;">
<table class="data-table">
<tbody>
<tr class="table-header">
<td>Name (Gender)</td>
<td>Correlation</td>
</tr>
<tr>
<td>Brody (M)</td>
<td>1.000</td>
</tr>
<tr>
<td>Dalilah (F)</td>
<td>0.992</td>
</tr>
<tr>
<td>Hayden (F)</td>
<td>0.991</td>
</tr>
<tr>
<td>Grayden (M)</td>
<td>0.990</td>
</tr>
<tr>
<td>Ailey (F)</td>
<td>0.989</td>
</tr>
<tr>
<td>Sahana (F)</td>
<td>0.989</td>
</tr>
<tr>
<td>Braylan (M)</td>
<td>0.988</td>
</tr>
<tr>
<td>Camila (F)</td>
<td>0.987</td>
</tr>
<tr>
<td>Jaslynn (F)</td>
<td>0.987</td>
</tr>
<tr>
<td>Rhylie (F)</td>
<td>0.986</td>
</tr>
</tbody>
</table>
</td>
<td style="border: 0px;">
<table class="data-table">
<tbody>
<tr class="table-header">
<td>Name (Gender)</td>
<td>Correlation</td>
</tr>
<tr>
<td>Dell (F)</td>
<td>-0.178</td>
</tr>
<tr>
<td>Leatha (F)</td>
<td>-0.178</td>
</tr>
<tr>
<td>Tempie (F)</td>
<td>-0.181</td>
</tr>
<tr>
<td>Vinnie (F)</td>
<td>-0.183</td>
</tr>
<tr>
<td>Parthenia (F)</td>
<td>-0.184</td>
</tr>
<tr>
<td>Helga (F)</td>
<td>-0.184</td>
</tr>
<tr>
<td>Nonie (F)</td>
<td>-0.186</td>
</tr>
<tr>
<td>Kittie (F)</td>
<td>-0.187</td>
</tr>
<tr>
<td>Minta (F)</td>
<td>-0.191</td>
</tr>
<tr>
<td>Retta (F)</td>
<td>-0.195</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<p>The names at right aren&#8217;t Biblical, but they do sound <em>really old</em>. They&#8217;re also all female names, which suggests that a better analysis would consider male and female names separately.</p>
<p>Code is available <a href="https://github.com/vedant/ssa-db-analysis">here</a>.<br />
Data is available <a href="http://www.ssa.gov/oact/babynames/limits.html">here</a>.</p>
]]></content:encoded>
			<wfw:commentRss>http://vedantmisra.com/2011/10/quantifying-trendy-names/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Why flash crashes happen</title>
		<link>http://vedantmisra.com/2011/08/why-flash-crashes-happen/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=why-flash-crashes-happen</link>
		<comments>http://vedantmisra.com/2011/08/why-flash-crashes-happen/#comments</comments>
		<pubDate>Fri, 19 Aug 2011 16:02:06 +0000</pubDate>
		<dc:creator>Vedant</dc:creator>
				<category><![CDATA[Research]]></category>
		<category><![CDATA[finance]]></category>
		<category><![CDATA[markets]]></category>

		<guid isPermaLink="false">http://vedantmisra.com/?p=185</guid>
		<description><![CDATA[The markets have been volatile recently. Various securities have experienced &#8220;mini flash crashes&#8221;: they suddenly lose a large fraction of their value, and regain it just as quickly. I wrote a piece for NECSI&#8217;s website to explain some of the market microstructure behind why flash crashes happen.]]></description>
			<content:encoded><![CDATA[<p>The markets have been volatile recently. Various securities have experienced &#8220;mini flash crashes&#8221;: they suddenly lose a large fraction of their value, and regain it just as quickly. I wrote <a href="http://necsi.edu/research/economics/whyflashcrashes.html">a piece for NECSI&#8217;s website</a> to explain some of the market microstructure behind why flash crashes happen.</p>
]]></content:encoded>
			<wfw:commentRss>http://vedantmisra.com/2011/08/why-flash-crashes-happen/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>The best (of the best)^4</title>
		<link>http://vedantmisra.com/2011/05/the-best-of-the-best4/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=the-best-of-the-best4</link>
		<comments>http://vedantmisra.com/2011/05/the-best-of-the-best4/#comments</comments>
		<pubDate>Thu, 05 May 2011 08:00:01 +0000</pubDate>
		<dc:creator>Vedant</dc:creator>
				<category><![CDATA[Article]]></category>
		<category><![CDATA[security]]></category>

		<guid isPermaLink="false">http://vedantmisra.com/?p=88</guid>
		<description><![CDATA[The Navy SEAL team that executed Operation Neptune&#8217;s Spear, the assault in Abbottabad last week in which Osama bin Laden was killed, has been referred to by the media as the &#8220;best of the best.&#8221; Let&#8217;s get a sense of how good they actually are. To get onto that team, you&#8217;d first have to join [...]]]></description>
			<content:encoded><![CDATA[<p>The Navy SEAL team that executed <a href="http://en.wikipedia.org/wiki/Death_of_Osama_bin_Laden">Operation Neptune&#8217;s Spear</a>, the assault in Abbottabad last week in which Osama bin Laden was killed, has been referred to by the media as the &#8220;best of the best.&#8221;  Let&#8217;s get a sense of how good they actually are.<br />
<span id="more-88"></span><br />
To get onto that team, you&#8217;d first have to join the Navy.  In <a href="http://en.wikipedia.org/wiki/United_States_Navy_SEAL_selection_and_training">SEAL training</a>, enlisted men and officers train side by side, so let&#8217;s suppose you take the &#8220;easy&#8221; route and skip Navy Officer Candidate School or four years at Annapolis and enlist.  To even get into boot camp, you&#8217;ll have to pass the &#8220;<a href="http://www.navy-prt.com/">Physical Readiness Test</a>,&#8221; which includes a 1.5 mile run, pushups, pullups, and a timed 500-yd swim test.  That gets you into Navy Basic Training, where for several weeks you&#8217;ll be running, swimming, and doing pushups.  Of those who sign an enlisted contract, <a href="http://www.navytimes.com/news/2007/04/navy_sealrecruiting_training_070408w/">some 58%</a> make it through Navy Basic Training.  Welcome to the Navy.</p>
<p>If you can score competitively on the SEAL Physical Screening test &#8212; that is, if you can swim 500 yards in 10 minutes, do around 80 pushups in 2 minutes, 80 sit-ups in 2 minutes, 11 pullups from a dead hang, and run 1.5 miles in boots in under 10:20 &#8212; you &#8216;ll be assigned to  BUD/S, the &#8220;Basic Underwater Demolition/SEAL&#8221; course.</p>
<p>BUD/S is a 28-week course comprised of three phases. First, a three week indoctrination program.  Then, seven weeks of physical conditioning that includes navigating the Pacific surf in inflatable boats, ocean swims, timed 4 mile runs in soft sand, and &#8220;hell week,&#8221; a 132-hour period of continuous physical activity (with 4 hours of sleep). Then an eight week diving course, and a ten week land warfare course.  Fewer than 20% of those who start BUD/S make it through.</p>
<p>Next, a 26-week course called &#8220;SQT,&#8221; Seal Qualification Training, consisting of Survival, Evasion, Resistance, and Escape training; tactical air operations; combat medicine; communications; cold weather mountaineering; combat swimming; land warfare; unarmed combat; and close weapons combat.  Most of those who make it through BUD/S also make it through SQT.</p>
<p>Welcome to the Navy SEALs, where you&#8217;ll be one of around 2,400 men in one of 9 teams, numbered 1-5 and 7-10.  Perform impressively over the next several years of deployments&#8212;roughly until your mid 30s&#8212;and you stand a chance of being recruited to the <a href="http://en.wikipedia.org/wiki/DEVGRU">Naval Special Warfare Development Group</a> (DEVGRU), AKA &#8220;Team Six.&#8221;  The selection process is classified.  If you consistently perform at the highest level while in the SEALs, you&#8217;ll enter a half-year training course.  Several SEAL operators have been killed during parachute accidents and   close-quarters battle training accidents in this phase of   training.  If you make it through, welcome to  Team Six.</p>
<p>You&#8217;re now a member of one of the most selective fighting forces in the world. Your unit responds directly to the President and to the  <a title="Joint Special Operations Command" href="http://en.wikipedia.org/wiki/Joint_Special_Operations_Command">Joint Special Operations Command</a>&#8216;s <a title="Special Mission Unit" href="http://en.wikipedia.org/wiki/Special_Mission_Unit">Special Mission Unit</a>.  But we&#8217;re not done yet.  Team Six consists of roughly <a href="http://en.specwar.info/special-forces/usa/devgru-seal-team-six/">200-400 operators</a> in three combat teams and three support teams.  You can&#8217;t end up in the support teams or on the two standard Assault teams.  You need to be in the Premier assault team, which consists of some 80 operators. You&#8217;ll have to distinguish yourself in armed combat and assault tactics to get into the Premier assault team.</p>
<p>Still not done.  The assault team in Operation Neptune&#8217;s Spear was 24 operators.  So you&#8217;ll need to distinguish yourself further within the Premier Assault Team to make the cut.</p>
<p>24 operators out of 79 Premier assault team members out of ~300 DEVGRU operators out of ~2,400 SEALs.  A selection rate of 1% after a selection rate of 20% (to get into the SEALs) after a selection rate of 50% (to enter the Navy).  Yikes.</p>
]]></content:encoded>
			<wfw:commentRss>http://vedantmisra.com/2011/05/the-best-of-the-best4/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Osama bin Laden and Bayesian Search</title>
		<link>http://vedantmisra.com/2011/05/osama-bin-laden-and-bayesian-search/?utm_source=rss&#038;utm_medium=rss&#038;utm_campaign=osama-bin-laden-and-bayesian-search</link>
		<comments>http://vedantmisra.com/2011/05/osama-bin-laden-and-bayesian-search/#comments</comments>
		<pubDate>Tue, 03 May 2011 12:00:08 +0000</pubDate>
		<dc:creator>Vedant</dc:creator>
				<category><![CDATA[Article]]></category>
		<category><![CDATA[math]]></category>
		<category><![CDATA[science]]></category>
		<category><![CDATA[security]]></category>

		<guid isPermaLink="false">http://vedantmisra.com/?p=35</guid>
		<description><![CDATA[A technique from operations research called Bayesian search was famously applied to finding vessels lost at sea by John Craven, the US Navy&#8217;s Chief Scientist for Special Projects. Given that the recent targeted killing of Osama bin Laden was the work of Navy Seals, it&#8217;s possible that directed belief networks were at some point used in [...]]]></description>
			<content:encoded><![CDATA[<p>A technique from operations research called <a href="http://en.wikipedia.org/wiki/Bayesian_search_theory">Bayesian search</a> was famously applied to finding vessels lost at sea by <a href="http://en.wikipedia.org/wiki/John_Pi%C3%B1a_Craven">John Craven</a>, the US Navy&#8217;s Chief Scientist for Special Projects. Given that the recent <a href="http://en.wikipedia.org/wiki/Targeted_killing">targeted killing</a> of <a href="http://en.wikipedia.org/wiki/Osama_bin_Laden">Osama bin Laden</a> was the <a href="http://www.nytimes.com/2011/05/02/world/asia/osama-bin-laden-is-killed.html?pagewanted=print">work of Navy Seals</a>, it&#8217;s possible that <a href="http://en.wikipedia.org/wiki/Bayesian_networks">directed belief networks</a> were at some point used in the search.<br />
<span id="more-35"></span><br />
To understand why Bayesian inference is useful in search, consider what you might do if you lost your ring after tripping on the curb.  Suppose your ring landed either in some nearby bushes or on the sidewalk.  Also, suppose that when you tripped, your arm stretched towards the bushes, so you think it&#8217;s more likely your ring is there than on the sidewalk.  Still, you&#8217;re likely to search the sidewalk first, because it&#8217;s easy.  That is, the probability of finding the ring given that it&#8217;s on the sidewalk is much higher than the probability of finding it if it&#8217;s in the bushes, even if it&#8217;s more likely to be in the bushes than on the sidewalk.  You&#8217;re considering not only where your target is likely to be, but also  how likely it is to find the target in each location.</p>
<p>Let&#8217;s look at how this intuition might have been used to find Osama bin Laden.</p>
<p>First, let&#8217;s formulate scenarios about bin Laden&#8217;s status.  Suppose that experts&#8217; opinions indicate that, with the probabilities shown, he&#8217;s either</p>
<ul>
<li>Able, and actively running Al-Qaeda: 20%</li>
<li>Able, but in hiding, and passively involved with Al-Qaeda: 30%</li>
<li>Receiving dialysis treatment in a hospital: 30%</li>
<li>Alive, but doing something else: 10%</li>
<li>Dead: 10%</li>
</ul>
<p style="text-align: left;">To decide where to search, we want to determine where he&#8217;s likely to be in each of these scenarios.  We&#8217;ll define a discrete search space by saying that we expect bin Laden to be in each of these places with the probabilities shown:</p>
<ul>
<li>Karachi: 10%</li>
<li>Islamabad: 10%</li>
<li>Lahore: 10%</li>
<li>Elsewhere in Pakistan: 50%</li>
<li>Outside of Pakistan: 20%</li>
</ul>
<p style="text-align: left;">Next we want to define conditional probabilities that he is in each of these locations given each of the above scenarios. For example, If he&#8217;s dead, finding him in, say, Islamabad, will be near impossible. In other words,</p>
<p style="text-align: center;"><img src='http://s.wordpress.com/latex.php?latex=P%28%20%5Ctextrm%7BIslamabad%7D%20%7C%20%5Ctextrm%7BDead%7D%20%29%20%5Capprox%200&#038;bg=ffffff&#038;fg=000000&#038;s=0' alt='P( \textrm{Islamabad} | \textrm{Dead} ) \approx 0' title='P( \textrm{Islamabad} | \textrm{Dead} ) \approx 0' class='latex' /></p>
<p>because his body may have already been buried or destroyed.  Consider what actually turned out to be the case &#8212; he was (effectively) in Islamabad, and was able, but in hiding.  Bayes&#8217; theorem tells us that</p>
<p style="text-align: center;"><img src='http://s.wordpress.com/latex.php?latex=P%28L%7CS%29%20%3D%20%5Cfrac%7BP%28S%7CL%29P%28L%29%7D%7BP%28S%29%7D%2C&#038;bg=ffffff&#038;fg=000000&#038;s=0' alt='P(L|S) = \frac{P(S|L)P(L)}{P(S)},' title='P(L|S) = \frac{P(S|L)P(L)}{P(S)},' class='latex' /></p>
<p>Where <img src='http://s.wordpress.com/latex.php?latex=L&#038;bg=ffffff&#038;fg=000000&#038;s=0' alt='L' title='L' class='latex' /> is a &#8220;Location&#8221;, and <img src='http://s.wordpress.com/latex.php?latex=S&#038;bg=ffffff&#038;fg=000000&#038;s=0' alt='S' title='S' class='latex' /> is a &#8220;Scenario.&#8221;  The probability of finding bin Laden in Islamabad, given that he&#8217;s able but in hiding, might then be:</p>
<p style="text-align: center;"><img src='http://s.wordpress.com/latex.php?latex=P%28%5Ctextrm%7BIslamabad%7D%20%7C%20%5Ctextrm%7BAbleHiding%7D%29%20%3D%20%5Cfrac%7B%28%5Ctextrm%7BAbleHiding%7D%7C%5Ctextrm%7BIslamabad%7D%29P%28%5Ctextrm%7BIslamabad%7D%29%7D%7BP%28%5Ctextrm%7BAbleHiding%7D%29%7D&#038;bg=ffffff&#038;fg=000000&#038;s=0' alt='P(\textrm{Islamabad} | \textrm{AbleHiding}) = \frac{(\textrm{AbleHiding}|\textrm{Islamabad})P(\textrm{Islamabad})}{P(\textrm{AbleHiding})}' title='P(\textrm{Islamabad} | \textrm{AbleHiding}) = \frac{(\textrm{AbleHiding}|\textrm{Islamabad})P(\textrm{Islamabad})}{P(\textrm{AbleHiding})}' class='latex' /></p>
<p style="text-align: center;"><img src='http://s.wordpress.com/latex.php?latex=%3D%5Cfrac%7B%280.4%29%280.1%29%7D%7B.3%7D%20%3D%200.13&#038;bg=ffffff&#038;fg=000000&#038;s=0' alt='=\frac{(0.4)(0.1)}{.3} = 0.13' title='=\frac{(0.4)(0.1)}{.3} = 0.13' class='latex' /></p>
<p style="text-align: left;">If we compute this probability for each location for each scenario, we have a set of probabilities that we can use to define a search path that starts at the point of highest probability and moves to lower probability areas.  If we had three locations, and three &#8220;likelihood classes&#8221; of finding bin Laden in each of them, we&#8217;d have a histogram like this one:</p>
<p style="text-align: center;">&nbsp;</p>
<div class="wp-caption aligncenter" style="width: 324px"><img src="http://reference.wolfram.com/mathematica/BarCharts/ref/Files/BarChart3D.en/O_1.gif" alt="" width="314" height="335" /><p class="wp-caption-text">Probabilities in discrete space.  One axis is &quot;Locations,&quot; the other is &quot;Likelihood of finding&quot;</p></div>
<p style="text-align: left;">Information we gather during the search might be used to revise probabilities during the search.  For instance, we might receive new intel while searching Lahore that decreases the probability that bin Laden is dead, which forces us to recompute all conditional probabilities in which we depended on <img src='http://s.wordpress.com/latex.php?latex=P%28%5Ctextrm%7Bdead%7D%29&#038;bg=ffffff&#038;fg=000000&#038;s=0' alt='P(\textrm{dead})' title='P(\textrm{dead})' class='latex' />.</p>
]]></content:encoded>
			<wfw:commentRss>http://vedantmisra.com/2011/05/osama-bin-laden-and-bayesian-search/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

