Intro and Data Management

1 Introduction This course will introduce you to Stata, a statistical software developed by StataCorp (stata.com). Stata is a statistical software for data analysis and data management. It has a fairly broad set of statistical features, data management facilities, and high-quality graphics. It includes Mata, a matrix programming language, although people who use matrix programming generally prefer other packages (e.g. Matlab, R) for this purpose. Stata is available in computers all over campus. If you decide to buy your own Stata license, buy at least the “Intercooled Stata” (IC), since the cheapest version “Small Stata” can handle only a small amount of observations and variables. In the “breath” vs “depth” trade-off, I’ll go for depth, since understanding really well the key features of Stata will allow you to learn other features on your own. It is really important that you get to a point where you can learn things on your own since new features are added with new releases, new types of analysis are possible, new commands become available, etc. 2 Opening Stata Stata has five main windows: Command, Output, Review, Variables, and Properties. The Command window is a window where you can type commands interactively. The Output window shows the output generated by those commands (a regression, a summary table, an error message, etc.). The Variables window shows the variables in the dataset as well as their labels. The Properties window shows the variable properties (label, type, format, etc.). Finally, the Review window stores all the commands that have been used in the current Stata session. If for any reason one of the windows doesn’t show on your screen, you may open it using the drop-down menus. For instance, Window\Command will open the Command window; Window\Variables will open the Variables window, and so on. In general, we use the command window only to test if something works, or what the output looks like. For most type of analysis, it is much better to use do-files. A do-file is a script that contains a set of Stata commands. The commands used in a do-file are identical to those used in the command window. A do-file is useful for replication, so anyone can replicate your work by re-running the file. In addition it is possible to “undo” any mistakes you may have made by correcting it in the do-file and re-running it over the original dataset, or to improve your analysis by adding some commands, etc. In this first lesson only we will act as if do-files didn’t exist. In this first lesson I’ll keep things as simple as possible to show you how you can perform simple data analysis in Stata without going into gory detail on technical stuff. In the next lesson I’ll start adding complexity. I postpone some technical details on command syntax, 1 Comments welcome. contact me at mbarron4 [at] ucsc 1 the use of help files to the next lecture. Today, I want you to have a quick feel of Stata so you can see what type of analysis you can produce with it. 3 Opening and saving Stata format datasets The use command allows to open a dataset, and the save command allows to save it. In this lesson we will focus on opening and saving Stata files (.dta), and in the next lesson we will learn to import and export other file formats (for example .csv, .txt, and .xlsx) In most cases you will want to save the dataset you opened under a different name. This will allow you to go back to the original dataset in case you made a mistake in the code. * PC . use "C:\Econ294A\original dataset.dta", clear [...stata commands...] . save "C:\Econ294A\modified dataset.dta", replace If you normally use a Mac, note the use of a slash “/” instead of a backslash “\” * Mac . use "/Users/manuel/Econ294A/original dataset.dta", clear [...stata commands...] . save "/Users/manuel/Econ294A/modified dataset.dta", replace The clear option clears any currently open dataset from memory, since Stata only handles one dataset at a time. The replace option is used to overwrite any pre-existing file under the same name. Stata comes preloaded with a number of example datasets. As the course progresses we will focus on real-world data, but these example datasets are a good starting point. You can access the preloaded datasets with the sysuse command. . sysuse dir /* <= shows all the pre-loaded datasets */ Let’s start by opening the auto dataset. It is a dataset that contains information on some makes and models, like price, mpg, headroom, trunk space, etc. If you are already somewhat familiar with Stata, you’ve probably used the “auto.dta” dataset before, since it seems to be the most commonly used dataset in Stata help files. . sysuse auto.dta, clear 2 As you can see, some variables (with their respective labels) appeared on the “variables” window. To see the actual dataset, open the browser by clicking on the Data Browser icon, or by typing . browse 4 Command syntax, abbreviations, case sensitivity We’ll get a bit deeper into command syntax in the next lecture, but by now, lets just say that Stata is case-sensitive. For example, the summarize command prints out summary statistics. After we open the “auto.dta” dataset, we can use it with the “price” variable, but not with the “Price” (note the capitalization) variable -since that variable doesn’t exist. For instance, summarize price prints out summary statistics of the price, but summarize Price will print out an error. Similarly, SUMMARIZE, Summarize, or other variations of the name will produce errors. . summarize price /* <= this is the right way */ Variable | Obs Mean Std. Dev. Min Max -------------+-------------------------------------------------------price | 74 6165.257 2949.496 3291 15906 . summarize Price /* <= the correct variable name is price, not Price */ variable Price not found r(111); . SUMMARIZE price /* <= the correct command name is summarize */ unrecognized command: SUMMARIZE r(199); 5 5.1 Logical and Mathematical Operators Mathematical Operators In Stata, the +, -, *, and / symbols denote addition, subtraction, multiplication, and division, respectively. The ˆ symbol is used to elevate to a power. If you want to use Stata as a calculator, you can’t use the numbers directly, you need to use the display command. Here are some examples: . 2 + 3 /* <= typing the formula directly doesn’t work. */ unrecognized command: 2 invalid command name . display 2+3 5 /* <= command */ /* <= Stata output */ 3 . display 2*4 8 . display 231/3 77 The default hierarchy of algebraic operations, unless you group operations with parentheses, is power, multiplication and division, addition and subtraction. . display 3-9*4+2^3 -25 . display (3-9)*4+2^3 -16 5.2 Logical Operators The logical operators in Stata are if, & (and) | (or). There are two equivalent ways for “not”: ! and ∼. To evaluate if a variable takes a specific value, we use two = symbols: “==”. You can also evaluate if a variable is different than (not equal to) “!=” or “∼=”; greater than “>”, greater than or equal to “>=”’, smaller than “<”’, and smaller than or equal to “<=”’. . count if foreign==1 22 . count if foreign==0 52 . count if foreign!=1 52 . count if foreign~=0 22 . count if foreign 22 . count if !foreign 52 In Stata (and other programming languages) one equal sign “=” is used to assign value, not to evaluate if a condition is met. Using one equal sign to evaluate if a condition is met will result in an error message. This is a common source of programming errors, especially for those of you who are new to coding. 4 . count if foreign = 1 =exp not allowed r(101); If we want two or more conditions to be met, we use the & symbol. . count if mpg<=20 & price<6000 24 . count if mpg<=20 & price<3000 0 If we want to evaluate whether at least one condition is met, we use the | symbol . count if mpg<=20 | price<3000 38 . count if foreign==1 | price>6000 36 If we are evaluating more than one condition using both & and |, it is best to use parentheses . count if (foreign==1 & price<3000) | (foreign==0 & price>5000) 23 5.3 Advanced use If you omit the equal signs after an if statement, Stata will evaluate whether the variable is greater than zero. As we will see later in the course, missing values are interpreted as infinity, so any observations with missing values are interpreted as larger than zero. This type of shorthand saves a few keystrokes but can land you in trouble if you are not sure of the implications. Its good practice to include a “non-missing” condition. . count if foreign 22 . count if !foreign 52 . count if rep78 74 . count if rep78 & !missing(rep78) 69 5 The last statement is equivalent to typing: . count if rep78>0 & rep78!=. 69 . count if rep78>0 & rep78<. 69 6 help Stata has very nice help files. To get help on a command, type help followed by the command name. If you are not sure of the command name, you can use findit (one word), which searches online for user-written commands, but also commands whose help files contain the word you are trying to find. findit is a command that allows searching for information on a topic across all sources, including the system help, the FAQs at the Stata website, the Stata Journal, and all Statarelated Internet sources including user-written commands. From findit, you can click to go to a source or to install additions. . help table The help file contains the following output: Title [R] table -- Flexible table of summary statistics Syntax table rowvar [colvar [supercolvar]] [if] [in] [weight] [, options] By clicking on [R] table (which appears blue on your screen) you’ll open the Stata Manual entry for table. The syntax shows you how to use the command. First, you need to write the command name “table”. After that, there’s the “rowvar”. This is a variable that will be tabulated in the rows of the table. Note the “[” symbol after “rowvar”. All the contents within the “[” symbol are optional. Optionally, you may type a colvar, which will give names to your table’s columns. You may add “if” and “in” conditions, as well as specify weights (we’ll see the in conditions and weights later in the course). If we scroll down, we’ll see the options available in this command: 6 Main contents(clist) contents of table cells; select up to five statistics; default is contents(freq) by(superrowvarlist) superrow variables Options cellwidth(#) csepwidth(#) stubwidth(#) scsepwidth(#) center left cell width column-separation width stub width supercolumn-separation width center-align table cells; default is right-align left-align table cells; default is right-align [...output omitted...] where elements of clist may be freq mean varname sd varname semean varname frequency mean of varname standard deviation standard error of the mean (sd/sqrt(n)) This means that we can issue a command like follows: table foreign, contents(mean price) 7 Elementary Data Analysis This section covers a set of commands that allows you to explore the data at hand. 7.1 describe describe produces a summary of the dataset at hand: its name, the number of observations, variables, size, among other dataset characteristics. It also prints the variable list, their storage type and display format, as well as their labels and value labels. If you are not familiar with any of these terms, don’t worry, we will talk about them as they become necessary. For a compact listing of the variable names in the dataset, we can use the “simple” option describe, simple. 7.2 summarize summarize prints out summary statistics for the numeric variable or variables requested. In its simple form, it prints the number of observations with valid data, the mean, standard deviation, minimum and maximum values in the dataset. 7 . summarize mpg Variable | Obs Mean Std. Dev. Min Max -------------+-------------------------------------------------------mpg | 74 21.2973 5.785503 12 41 summarize, detail provides a bit more detail: the median and some of the percentiles, as well as the variance, skewness and kurtosis. Try it. . summarize mpg, detail Mileage (mpg) ------------------------------------------------------------Percentiles Smallest 1% 12 12 5% 14 12 10% 14 14 Obs 74 25% 18 14 Sum of Wgt. 74 50% 75% 90% 95% 99% 20 25 29 34 41 Largest 34 35 35 41 Mean Std. Dev. 21.2973 5.785503 Variance Skewness Kurtosis 33.47205 .9487176 3.975005 summarize on its own will produce summary statistics of all the variables in the dataset: . summarize Variable | Obs Mean Std. Dev. Min Max -------------+-------------------------------------------------------make | 0 price | 74 6165.257 2949.496 3291 15906 mpg | 74 21.2973 5.785503 12 41 rep78 | 69 3.405797 .9899323 1 5 headroom | 74 2.993243 .8459948 1.5 5 -------------+-------------------------------------------------------trunk | 74 13.75676 4.277404 5 23 weight | 74 3019.459 777.1936 1760 4840 length | 74 187.9324 22.26634 142 233 turn | 74 39.64865 4.399354 31 51 displacement | 74 197.2973 91.83722 79 425 -------------+-------------------------------------------------------gear_ratio | 74 3.014865 .4562871 2.19 3.89 foreign | 74 .2972973 .4601885 0 1 8 You may have noticed that the “make” variable appears to have 0 observations. However, if you browse through the dataset, you will see that it has 74 valid observations. The reason is that “make” is a string variable (letters, not values), and it does not make sense to produce summary statistics of string variables. 7.3 count count prints out the number of observations. Without any other parameters, it counts the number of observations in the dataset. Most commonly, we use it to count the number of observations that fulfill some condition. . count 74 . count if mpg<=20 38 . count if mpg>3500 0 7.4 tabulate tabulate, abbreviated by tab, is a command that outputs simple or 2x2 tabulations. It takes either one or two variable names as arguments. The first variable determines the row names, and the second the column names. . tab headroom Headroom | (in.) | Freq. Percent Cum. ------------+----------------------------------1.5 | 4 5.41 5.41 2.0 | 13 17.57 22.97 2.5 | 14 18.92 41.89 3.0 | 13 17.57 59.46 3.5 | 15 20.27 79.73 4.0 | 10 13.51 93.24 4.5 | 4 5.41 98.65 5.0 | 1 1.35 100.00 ------------+----------------------------------Total | 74 100.00 . tab headroom foreign 9 Headroom | Car type (in.) | Domestic Foreign | Total -----------+----------------------+---------1.5 | 3 1 | 4 2.0 | 10 3 | 13 2.5 | 4 10 | 14 3.0 | 7 6 | 13 3.5 | 13 2 | 15 4.0 | 10 0 | 10 4.5 | 4 0 | 4 5.0 | 1 0 | 1 -----------+----------------------+---------Total | 52 22 | 74 With tabulate you can produce quick summary statistics for another variable at each value of headroom. For instance, . tab headroom, summarize(price) Headroom | Summary of Price (in.) | Mean Std. Dev. Freq. ------------+-----------------------------------1.5 | 5,509.5 1,009.613 4 2.0 | 4,822.846 854.11424 13 2.5 | 6,591.571 3,196.544 14 3.0 | 5,906.231 3,403.979 13 3.5 | 7,580.933 4,086.414 15 4.0 | 6,458.5 2,256.714 10 4.5 | 5,018 926.56606 4 5.0 | 4,060 0 1 ------------+-----------------------------------Total | 6,165.257 2,949.496 74 . tab foreign, summarize(turn) | Summary of Turn Circle (ft.) Car type | Mean Std. Dev. Freq. ------------+-----------------------------------Domestic | 41.442308 3.9675817 52 Foreign | 35.409091 1.5010819 22 ------------+-----------------------------------Total | 39.648649 4.3993537 74 10 7.5 table table is the command I use most frequently. It has the most flexibility (as we saw in the help file). . table headroom foreign -----------------------------Headroom | Car type (in.) | Domestic Foreign ----------+------------------1.5 | 3 1 2.0 | 10 3 2.5 | 4 10 3.0 | 7 6 3.5 | 13 2 4.0 | 10 4.5 | 4 5.0 | 1 -----------------------------. table headroom foreign, c(mean price) -----------------------------Headroom | Car type (in.) | Domestic Foreign ----------+------------------1.5 | 5,269.7 6,229 2.0 | 4,553.9 5,719.3 2.5 | 6,354.3 6,686.5 3.0 | 6,222.1 5,537.7 3.5 | 7,440.7 8,492.5 4.0 | 6,458.5 4.5 | 5,018 5.0 | 4,060 ------------------------------ 7.6 tabstat tabstat is another useful command to produce summary statistics. It allows the user to define the variables of interest as well as the statistics of interest. In addition, you can choose to see the output with the variables as rows or columns. Here are some examples: . tabstat price headroom weight, s(mean median p95) stats | price headroom weight 11 ---------+-----------------------------mean | 6165.257 2.993243 3019.459 p50 | 5006.5 3 3190 p95 | 13466 4.5 4290 ---------------------------------------. tabstat price headroom weight, s(mean median p95) columns(statistics) variable | mean p50 p95 -------------+-----------------------------price | 6165.257 5006.5 13466 headroom | 2.993243 3 4.5 weight | 3019.459 3190 4290 -------------------------------------------- 8 Using these results Say we want to “demean” the trunk variable in the auto dataset. Demeaning is subtracting the mean from a variable, and is an easy way of simplifying calculations in a number of statistical methods. . sum trunk Variable | Obs Mean Std. Dev. Min Max -------------+-------------------------------------------------------trunk | 74 13.75676 4.277404 5 23 . g dmtrunk = trunk - 13.75676 . sum dmtrunk Variable | Obs Mean Std. Dev. Min Max -------------+-------------------------------------------------------dmtrunk | 74 -3.21e-06 4.277404 -8.75676 9.24324 The mean of dmtrunk is -3.21e-06, which means -0.00000321. This essentially zero, as we wanted. In the programming section of the course we will learn how to store the results of these commands in memory. This will allow us to use them in an automated way, without the need to type the numbers. 12 9 Generating variables generate, usually abbreviated as gen or simply g lets the user create new variables in the dataset, and as such it is one of the most commonly used Stata commands. Let say we are particularly interested in cheap cars, defining cheap as those that cost less than $4,500. . generate cheap = 1 if price < 4500 (48 missing values generated) . summarize cheap Variable | Obs Mean Std. Dev. Min Max -------------+-------------------------------------------------------cheap | 26 1 0 1 1 As you can see, Stata generated 26 “ones” 48 “missing values”. A missing value means that Stata doesn’t know what value should that variable take. Simply put, this happened because we told Stata what to do if the price was less than $4,500, but not what to do if the price was greater than that. To do that, we need to tell Stata how to replace those missing variables. If we try to use the generate command again, we will get an error message, since the “cheap” variable already exists we can’t generate it again. What we can do is replace some of its values. In this case we will replace the missing values “.” with zeros. . generate cheap=0 if price>=4500 cheap already defined r(110); . replace cheap=0 if price>=4500 (48 real changes made) . sum cheap Variable | Obs Mean Std. Dev. Min Max -------------+-------------------------------------------------------cheap | 74 .3513514 .4806512 0 1 Before we generate another variable called “cheap” we need to drop the existing “cheap” variable. We do that with the drop command. . drop cheap [when Stata drops a variable it generates no output] . drop cheap variable cheap not found 13 As you can see, we can’t drop a variable twice. After we drop it, Stata has nothing on memory called “cheap”, and since it can’t drop something that doesn’t exist, it returns an error message. Let’s get that variable back, but let’s take a shortcut. In every software package, there is a variety of ways of doing the same thing. At the beginning most of you may be more comfortable with the way we just saw (which is safer). Here is a shortcut: . g cheap=price<=4500 . g expensive=price>4500 This tells Stata to generate a variable called “cheap”, that will take the value of 1 if the condition is met, and zero otherwise. We also created an “expensive” variable that takes the value of 1 if price is above $4,500. Note that this method is shorter, but riskier: we need to be 110% sure that the complement to our conditions are what we think we are. In this toy example, they are: all prices above $4,500 are indeed above $4,500 and viceversa. However, lets introduce a feature of real-world data to our toy example. Lets say we are missing the price for two observations. To simulate this problem, type: . replace price=. if make=="Chev. Monza" (1 real change made, 1 to missing) . replace price=. if make=="Chev. Impala" (1 real change made, 1 to missing) Now, lets drop “cheap” and “expensive” and lets create them again. . drop cheap . drop expensive . g cheap=price<=4500 . g expensive=price>4500 Stata interprets missing values as infinity. Since infinity is greater than 4500, both the Monza and the Impala will be classified as expensive cars. To avoid that, it is usually safer to add a condition: . g cheap=price<=4500 if price<. (2 missing values generated) . g expensive=price>4500 if price!=. (2 missing values generated) As we see, Stata generated two missing values in each case, corresponding to our makes for which we deleted the prices. 14 10 Drop-down menus I do not know how to use the drop-down menus, except to generate graphs. In its last few versions, Stata has become more and more “drop-down menu’ friendly, but when I started using Stata back in 2000 (Stata 5.0), there were very few drop-down menu options. I mostly use drop-down menus to generate graphs. The nice feature of Stata is that after you click through its drop-down menus, Stata prints the code required for the graphs in the output window, so you can see the commands (and copy and paste them in your do-file). When I forget about the command for a particular graph, I click through the windows, generate the basic version, and then tweak the options we’ll see in the following lectures to change its appearance. 11 Graphs Stata has a fairly extensive set of options to generate graphs. We will see a number of them along the course. In this lesson we will start with the most basic graphics. The most common way to generate graphs is by using the drop-down menus. When you do that, Stata generates the code and writes it in the output window. Twoway plots can be created by clicking on Graphics/two-way graphs/ create... and then selecting the desired options in the dialogue window. For instance, select “basic plots”, “scatter”, price as the “y variable”, and mpg as the “x variable”. Then hit accept, then ok, and you should see a scatterplot of the variables you picked. More importantly, you will see a line of code that would have generated that output in the output window. This is a nice feature of Stata that facilitates the transition from point-and-click to writing commands. 11.1 Scatter Plots 0 5,000 Price 10,000 15,000 . twoway (scatter price mpg) 10 20 30 40 Mileage (mpg) 15 11.2 Linear Fit To plot the linear fit between two variables, we use the folioing command: 2000 Fitted values 4000 6000 8000 . twoway (lfit price mpg) 10 20 30 40 Mileage (mpg) 11.3 Overlaying Graphs We can overlay two (or more) lines in the same graph by calling two-way and typing the code for each graph in parentheses. Optionally, you can write || between the code fore each graph. Let’s see how that works: 0 5000 10000 15000 . twoway (lfit price mpg) || (scatter price mpg) 10 20 30 40 Mileage (mpg) Fitted values Price 16 11.4 Tweaking Graph Appearance In following lectures I’ll cover some easy tricks to improve how your graphs look. From titles, to line color, pattern, or width, type of markers, scale, etc. 17