How Open Source Software Works

AGENDA What is open source and why do people participate • How does open source work? • What motivates developers? How does work get done? WHAT IS OPEN SOURCE? Firm-based software development Open Source software development Code developed by firm Code developed by community Code locked via binary and sent to customers Code distributed to users Customers run program Users create binary Bug complaints & feature request to company Users run program ! Users find and fix bugs, and create new features (Oops) Users distribute modifications ! (Patch) OPEN SOURCE PRINCIPLES Intellectual property Code should always be open “Free speech, not free beer” Development paradigm Resource model Extensive involvement of user/developer community Good ideas come from solving a problem or scratching an itch “Release early, release often” “The three obligations: to give, to receive, to reciprocate” C “Copyleft” C C “Use copyright to ensure copyleft” Modularize code Peer leadership vision, engagement, code HOW DOES OPEN SOURCE WORK? Virtual teams Provide “home” for code Establish team norms e od c t Ge Users Self organize Use software Do Get na co te de co de New releases Coordinate/ Influence User developers Leadership Report bugs Support each other Join Scratch coding itch, solve problem Initial code base and vision Implement features & fix bugs Own and modularize kernel Provide expertise Licensing Proselytize movement Prepackage open source For-profit companies OEM open source “Liberate” code Provide support and consulting LINUX DESIGN IS MORE EMERGENT THAN DIRECTED Excerpts from Postings by Linus Torvalds Rik van Riel: “It seems like Linux really isn't going anywhere in particular and seems to make progress through sheer luck” Linus (in several emails in a longer thread): “Hey, that's not a bug, that's a FEATURE! [his emphasis] “Do I direct some stuff? Yes. But, quite frankly, so do many others. Alan, Al, David, even you. And a lot of companies are part of the evolution whether they realize it or not. And all the users end up being part of the ‘fitness testing’.... “A strong vision and a sure hand sound good on paper. It's just that I have never met a technical person (including me) whom I would trust to know what is really the right thing to do in the long run.... “Too strong a strong vision can kill you-- you'll walk right over the edge firm in the knowledge of the path in front of you... “I'd much rather have ‘brownian motion,’ where a lot of microscopic directed improvements end up pushing the system slowly in a direction that none of the individual developers really had the vision to see on their own.” MORE THAN 85,000 MESSAGES A MONTH COORDINATE THE LINUX ENTERPRISE User Development Corporate mailing lists redhatlist suselinux-e suse-security debian -devel -changes Community mailing lists linux-newbie suse-linux Extensions debian devel linux “beer hiking club” linux-raid linux-kernel debian -user alsa-devel linux.redhat.misc linux. redhat. install Corporate bulletin boards alt.os.linux. mandrake comp.os.linux. advocacy comp.os. linux. networking alt.os.linux comp.os. linux. hardware comp.os. linux.misc Community bulletin boards 1,000 Posts/month CRITICAL TO UNDERSTAND MOTIVATIONS AND ORGANIZATIONAL STRUCTURE OF OPEN SOURCE Motivations • Are developers working for “free”? • Why are they participating? • What kind of effort are they contributing? Organizational Structure: • How are projects organized? • What kind of structure enables dispersed, virtual collaboration? • What are the lessons for firms? Is the open source movement a fad or is it sustainable? AGENDA What is open source and why do people participate • How does open source work? • What motivates developers? How does work get done? SURVEY OF PROJECTS ON SOURCEFORGE.NET TO UNDERSTAND MOTIVATIONS IN COMMUNITY SourceForge II SourceForge Target Process 10% random selection of alpha, beta, production projects (1); 1648 developers identified Mature projects with two or more developers; 573 developers identified Response Sent personal email link to web-based survey Received 526 responses; 34% res. rate Sent personal email link to web-based survey Received 169 responses; 30% response rate Results based on 684 usable responses (1) Projects had 50% or greater activity level OVERVIEW OF KEY FINDINGS ON HACKER MOTIVATIONS Why should we care? High creativity What motivates hackers? ? Fun, skill, freedom and need Increasing knowledge biggest benefit Losing sleep biggest cost 8 7 6 5 Who are these guys? 4 3 2 1 Volunteer significant time IT professionals 52 49 46 43 40 37 34 31 28 25 22 19 16 13 0 Generation Xers What about the community? Strong identification Global effort Peer leadership preferred - 12 - OSS PROJECTS AND PROGRAMMING TURNS ON HACKERS 61.7% “This project is as (or most) creative as anything I have done” 72.6% “When I program, I lose track of time” “Like composing 48.4% poetry or music” 60.0% “With one more hour in the day, I would spend it programming” Note: “...like composing poetry...” answer chosen as one of top three attitudes by participants; other answers based on degree of participant agreement with statement - 13 - ? OVERALL HACKER MOTIVATIONS Intellectually stimulating 44.9 Improves skill 41.3 Work functionality 33.8 Code should be open 33.1 Non-work functionality 29.7 Obligation from use 28.5 Work with team 20.3 17.5 Professional status 16.3 Other Open Source reputation 11.0 Beat proprietary software 11.1 License forces me to 0.2 0 10 20 30 40 50 Percent of respondents Note: Question asked for top three motivators of F/OSS participation, n=684 - 14 - VOLUNTEER CONTRIBUTORS MAKE UP MAJORITY OF RESPONDENTS Volunteer Paid 60 40 “Have you been financially compensated in any way for participating in this project?” No Yes “Is your direct supervisor aware of your project participation (during work time)?” No Yes Percent of responses Selection criteria - 15 - ? MOTIVATIONS DIFFER BETWEEN PAID AND VOLUNTEER CONTRIBUTORS 46.6 Intellectually stimulating 40.8 46.2 Improves skill 30 34.8 Code should be open 29.1 34.5 Non-work functionality 18.7 21.8 Work functionality 62 28.7 28.1 Obligation from use 22.5 Work with team 15.3 15.3 Professional status 22.6 Volunteers 11.1 10.8 Open Source reputation “Paid” (1) to contribute 11.1 11.8 Beat proprietary software 0 10 20 30 (1) Includes those working on F/OSS project full time, part time, and those sanctioned by supervisors (2) Volunteers= 479, paid=205 Note: Question asked for top three motivators of F/OSS participation, n=684 40 50 60 70 Percent of respondents - 16 - MOTIVATIONS AND CONTRIBUTION STATUS SEGMENT HACKERS “Community Believers” (19%) “Hobbyists” (27%) Motivations Do it because they feel obligation and believe source code should be open “Professionals” (25%) Do it for work need ? Do it for non-work “Learning & Stimulation” (29%) Do it for skill improvement and fun - 17 - OPEN SOURCE IS A GENERATION “X” PHENOMENON Average Age: 30 Years Percent of respondents 7 Average age: 30 Gen Y 13.7% 6 Gen X 70.2% Boomers 16.1% Volunteers: 29 Paid: 32 5 4 3 2 1 0 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 55 Note: n = 677 total responses Age And 98% male - 18 - OPEN SOURCE IS A GLOBAL ENTERPRISE Venezuela 1 Argentina 3 Hungary 4 Brazil 9 Vancouver Toronto Ottawa 9 8 3 SF Bay Area14 Boston 10 Denver 10 Los Angeles 10 Atlanta 6 Austin 6 New York 6 Baltimore 5 Kansas City 5 Portland 5 Seattle 5 St. Louis 5 Washington 5 Columbus 4 Detroit 4 Milwaukee 4 Philadelphia 4 San Diego 4 Dallas 3 Houston 3 Indianapolis 3 Pittsburgh 3 Phoenix 3 Salt Lake City3 Chicago 2 Lexington 2 Canada 39 U.S. 267 Americas 46.9% Note: n = 684 total responses, ROW = Rest of the World Montreal 2 Calgary 1 Quebec City 1 Madison 2 Minneapolis 2 Nashville 2 Providence 2 Sacramento 2 Tampa 2 Tulsa 2 Ames 1 Ann Arbor 1 Bozeman 1 Charlotte 1 Cincinnati 1 Cleveland 1 Ft. Lauderdale1 Gainesville 1 Hartford 1 Huntsville 1 Lansing 1 Louisville 1 New Haven 1 New Orleans 1 Orlando 1 Richmond 1 San Antonio 1 Syracuse 1 Lithuania 1 Latvia 1 Ireland 1 Iceland 1 Estonia 1 Croatia 1 Bulgaria 1 Belarus 1 Austria Spain 55 Denmark 56 Spain 7 6 Switzerland Belgium 8 Belgium 6 Switzerland Austria 6 10 Norway Norway 11 9 Italy Italy 15 11 Sweden Sweden 15 12 New Zealand 4 Russia 2 Portugal 2 Netherlands 19 25 Sydney 9 Canberra 5 Melbourne 5 Brisbane 2 Queensland1 7 6 5 5 4 3 Europe 42.4% Gabon 1 Armenia 1 Israel 3 Finland 2 U.K. 45 33 Germany 77 62 Morocco 1 Poland 2 Czech Rep. 2 Australia 42 Slovenia 3 Munich Munich 7 Berlin Berlin 6 Frankfurt 5 Frankfurt Stuttgart Stuttgart 5 Nuremberg Nuremberg4 Hamburg 3 Hamburg South Africa 1 Angola 1 Slovak Rep. 2 France 17 25 London London 16 16 Leeds 4 4 Leeds Bristol 2 2 Bristol Manchester 2 2 Manchester Edinburgh Edinburgh1 1 Taiwan 1 South Korea 1 Singapore 1 Malaysia 1 Japan 1 Indonesia 1 Hong Kong 1 China 2 Aachen Aachen 2 Dusseldorf Dusseldorf2 Heidelberg2 Heidelberg Cologne Cologne 1 Hannover Hannover 1 Leipzig 1 Leipzig 2 2 2 1 1 1 India 8 Greece 3 Romania 4 ROW 10.7% - 19 - RESPONDENTS VOLUNTEER A LOT OF TIME “This” project All projects Overall mean=7.5 hours/ week Volunteers=5.8 hours paid= 11.4 hours Overall mean= 14.09 hours/ week Volunteers=13.5 hours paid= 20.9 hours 40 Percent of respondents 35 35 30 30 25 25 20 20 15 15 10 10 5 5 0 0 0 1 to 5 6 to 12 13 to 20 21 to 40 >40 Hrs/week 0 1 to 5 6 to 12 13 to 20 21 to 40 >40 Hrs/week Note: n=684 - 20 - CONTRIBUTE TO MANY PROJECTS Current project All projects Mean = 2.6 Volunteer = 2.4 paid = 3.0 Mean = 4.9 Volunteer = 4.5 paid = 5.8 35 Percent of respondents 30 35 25 25 20 20 15 15 10 10 5 5 0 0 30 0 1 2 3 4+ Number of projects 1 2 3 4 5 6+ Number of projects Note: N = 684 - 21 - PARTICIPANTS ARE MOSTLY EXPERIENCED IT PROFESSIONALS Current occupation Percent of respondents 50 45.4 45 40 35 30 25 19.6 20 14.8 15 10 6.3 7.0 6.3 5 0 Programmer Sys. Admin. IT Manager Student Other Academ. Average 11 years of programming experience Note: n=678 - 22 - PROJECT CREATIVITY LARGEST DRIVER OF EFFORT Regression on Project Hours/ Week What is significant? What is not? + Creativity on project Age + Professional status (1) IT Job - IT training (1) Hacker affiliation Founder of project Prior social connection USA based Work functionality Non-work functionality Intellectually stimulating Improves skill Work with team Code should be open Beat proprietary software Community reputation Obligation from use (1) Volunteers only - 23 - HIGH PROJECT CREATIVITY DRIVES HOURS CONTRIBUTED Volunteer Paid Average hours/ week contributed 5.8 11.4 Impact of unit change in creativity (scale: 1- much less, 2-somewhat less, 3-equally, 4-most creative) 3.3 6.3 Anticipated hours with one unit increase in creativity 9.1 17.7 Percent increase in hours 57% 55% - 24 - SUMMARY OF COMMUNITY MOTIVATIONS No single motivation driving community participation • Community is a “Big Tent” – participants can contribute for any reason • Professional needs, community motivation, fun and learning and hobby are the primary types of motivations for participation Negativity towards commercial software developers not a prime mover Feeling creative biggest predictor of incremental effort on projects - 25 - AGENDA What is open source and why do people participate • How does open source work? • What motivates developers? How does work get done? - 26 - OPEN SOURCE COMMUNITIES FACE UNIQUE CHALLENGES “Voluntary” participation Dispersed contributors Part-time and intermittent participation Lean communication & collaboration technology Rudimentary management tools “Missing” project managers So How Do They Get Work Done? - 27 - Research Questions For My Dissertation What are the specific practices for distributed development used by Open Source communities? How do these practices work and how are they inter-related? How similar and different are the practices used by Open Source projects to firm-based development? - 28 - Dissertation Analyzes Practices From Two Cases Characteristics PostgreSQL Cocoon Technical • Area Database XML web dev. environment • Rate of change in technology Stable – mature technology Fast changing & new • Lines of code ~500, 000 LOC ~1,000,000 LOC • End-users Database administrators Web developers • Competition Oracle, IBM, Microsoft, MySQL ??? • Origins UC Berkeley academic project User need from Apache project • Time active (1984-1996)1996 2000 • Steering committee size 5 32 • Commit access 12 60 • Development list size 3,039 1689 • Affiliation None Apache Software Foundation “Organization” - 29 - Grounding The Model In Actual Data: Building A Process History Of Each New Feature In PostgreSQL (7.3 -7.4, November 2002-2003) Major Changes Detailed Changes • 17 major changes between 7.3 – 7.4 (as seen in release notes) • 375 detailed changes •140 detailed changes constitute major changes CVS Commits • ~1000 CVS commits • CVS commit info = Committer, Date/Time. Location, LOC+/-, Files& Comments E-mail Messages •“Hackers” (16,797 messages, 3,332 threads, 798 authors) •“Patches” (3,332 messages, 806 threads, 188 authors) Follow-up with E-mail, telephone, IRC and face-to-face interviews - 30 - Study Design And Data Inductive grounded theory building of Open Source development process • Virtual ethnography of two communities (PostgreSQL & Apache Cocoon) – 8 months • Extensive interviewing of project participants • Analysis of e-mail and source code change archives • Analyzing one year’s worth of complete technical activity (~15,000 e-mails and ~2000 source code changes) for each project to build innovation history of each new feature • Have developed a preliminary model of development practices - 31 - User/Developer(s) Creates New Feature Mario Weilguni (user/developer) outlines a problem that he faces with database vacuuming (Sept 3, 2002) 9 other individuals participate in the discussion about this need and possible solutions – general agreement that it would be good to do – Tom Lane (steering committee) makes several technical suggestions (Sept 3, 2002) Shridhar Dahitankar (new user/developer) announces that he is going to work on this but needs some information (Sept 3, 2002) Matthew O’Connor (new user/developer) also announces that he is going to work on the same topic (Sept 3, 2002) Shridhar Dahitankar – announces that the code is ready based on prior discussion (Sept 23, 2002) Other people give feedback and Mario reveals an attempt at the same feature (Sept 24, 2002) Matthew O’Connor asks some coding questions (September 24, 2002) Matthew O’Connor announces a “new & improved” version of the Shridhar’s code (November 26, 2002) Shridar Dahitankar asks questions and makes suggestions (November 27-28, 2002) - 32 - Preliminary Inductive Model Of Open Source Practices That Substitute For “Standard Management” Practices Minimal Verifiability Enabling Conditions Modularity Low Cost Reversibility 4 Action Before Explicit Coordination Legend 3 Optimistic Concurrency Micro Contribution 2 Task SelfSelection Community Need User need 1 Open for “all” Transparent Knowledge Broadcast 5 Project Repository: (Code, Bugs, Email, Wiki) 6 Lazy and Distributed Evaluation 7 8 Lazy Consensus High Energy Resolution Decision Escalation Continuous Integration - 33 - Many Thanks To: Eric von Hippel Wanda Orlikowski Leslie Perlow Carliss Baldwin Georg von Krogh Stefano Mazzocchi Ben Hyde Brian Behlendorf Neil Conway Luis Villa Bruce Momijian Guido van Rossum Anthony Baxter LMU-MIT User Innovation Workshop MIT Sloan Doctoral Seminar in Strategy + numerous participants in Python, PostgreSQL, Apache, and Freenet projects - 34 -