Readings in Optimization Presentation 5

Structure of the talk 1. Introduction to polytopes, linear programming, and the simplex method. 2. A few facts about polytopes. 3. Choosing the next pivot. Main result in this talk. 4. Subexponential randomized simplex algorithms. 5. Duality between two subexponential simplex algorithms. 6. The Hirsch conjecture, and applying randomized simplex to it. 7. Improving diameter results using an oracle for choosing pivots. 1 Polytopes and polyhedra A polyhedron P ⊆ Rd is the intersection of finitely many halfspaces, or in matrix notation P := {x ∈ Rd : Ax ≤ b}, where A ∈ Rn×d and b ∈ Rn. A polytope is a bounded polyhedron. Dimension of polyhedron P is dim(P ) := dim(aff(P )), where aff(P ) is the affine hull of all points in P . A polyhedron P ∈ Rd with dim(P ) = k is often called a k-polyhedron. If d = k, P called full-dimensional. (Most of the time we assume full-dimensional d-polyhedra, not concerned much about the surrounding space.) An inequality ax ≤ β, where a ∈ Rd and β ∈ R, is called valid if ax ≤ β for all x ∈ P. 2 Vertices, edges, ..., facets A face F of P is the intersection of P with a valid inequality ax ≤ β, i.e. F := {x ∈ P : ax = β}. Faces of dimension d − 1 are called facets, 1 ... edges, and 0 ... vertices. Vertices are points, ⇔ basic feasible solutions (algebraic), or extreme points (linear cost). Since 0x ≤ 0 is valid, P is a d-dimensional face of P . 0x ≤ 1 is valid too, so ∅ is a face of P , and we define its dimension to be −1. Some vertices are connected by edges, so we can define a graph G = (V (G), E(G)), where V (G) = {v : v ∈ vert(P )} and E(G) = {(v, w) ∈ V (G)2 : ∃ edge E of P s.t. v ∈ E, w ∈ E}. For unbounded polyhedra often a ∞ node is introduced in V (G), and we add graph arcs (v, ∞) whenever v ∈ E where E is an unbounded edge of P . 3 Example of a 3-polytope 6 8 2 4 2 4 6 8 F F 5 7 5 1 7 3 1 3 Figure 1: A 3-polytope (left) and its graph (right). Four vertices, three edges, and facet F are shown in corresponding colors. 4 Linear programming and the simplex method A linear programming problem max{cx : Ax ≤ b} is the problem of maximizing a linear function over a polyhedron. • If problem bounded (cost of feas. sol. finite), optimum can be achieved at some vertex v. • If problem unbounded, can find edge E of P = {x ∈ Rd : Ax ≤ b} s.t. cx is unbounded on the edge. • If problem bounded, vertex v is optimal ⇔ cv ≥ cw for all w adjacent to v (for all (v, w) ∈ E(G)). Geometrically, the simplex method starts at a vertex (b.f.s.) and moves from one vertex to another along a cost-increasing edge (pivots) until it reaches an optimal vertex (optimal b.f.s). 5 Vertices as intersections of facets Any polytope can be represented by its facets P = {x ∈ Rd : Ax ≤ b}, or by its vertices P = conv({v : v ∈ vert(P )}). If vertices are given, then LP is trivial—just select the best one. Most of the time, facets are given. Number of vert. exponential in number of facets makes generating all vertices from the facets impractical. Represent a vertex v as intersection of d facets. Any vertex is situated at the intersection of at least d facets; any non-empty intersection of d facets yields a vertex. When situated at a vertex v given by ∩di=1Fi, easy to find all adjacent vertices. Remove each facet Fi, and intersect with all other facets not in {F1, . . . , Fd}. Except when... 6 Degeneracy and simple polytopes When a vertex is at the intersection of > d facets, procedure above may leave us at the same vertex. Worse, sometimes need such changes before can move away from a vertex in cost-increasing direction. This is (geometric) degeneracy. In standard form degenerate vertices yield degenerate b.f. solutions. Other “degenerate” b.f. solutions may appear because of redundant constraints. If all vertices of P belong to at most d facets (⇒ exactly d), P is called simple. Simple polytopes correspond to non-degenerate LPs, and have many properties [Zie95, Kal97]. We restrict ourselves to simple polytopes. Ok for two reasons: 1) any LP can be suitably perturbed to become non-degenerate; 2) perturbation can be made implicit in the algorithms. 7 A few facts about polytopes Disclaimer: results not used or related to subexponential simplex pivot rules (main result in this talk). The f -vector: fk (P ) :=# of k-faces of P . Degrees: let degc(v) w.r.t. to some objective function c be the # of neighboring vertices w with cw < cv. The h-vector: hk,c(P ) :=# of vertices of degree k w.r.t. objective c in P . Note: there is always one vertex of degree d, and one of degree 0. Property: hk,c(P ) = hk (P ), independent of c. 8 hk,c(P ) = hk (P ), proof (1/2) Proof. Count p := |{(F, v) : F is a k-face of P , v is max. on F }, in two ways. Pick facets. p = fk (P ). Because c in general position ⇒ v unique for each F , hence On the other hand, pick a vertex v, and assume degc(v) = r. Let T = {(v, w) : cv > cw}, by definition |T | = r. For simple polytopes, each vertex v has d adjacent edges, and any k of them define a k-face F that includes v. � � � � |T | r So, # of k-facets that contain v as local maximum is = . k k 9 hk,c(P ) = hk (P ), proof (2/2) Summing over all v ∈ vert(P ), we obtain fk (P ) = � � r h (P ) . r=k r,c k �d Equations linearly independent in hr,c. This completely determines hr,c(P ) in terms of fk (P ). But fk (P ) independent of c, so same true for hr (P ). 10 The Euler Formula and Dehn-Sommerville Relations � � �d r r−k We can expess hk (P ) = r=k (−1) fr (P ) . k We know that h0(P ) = hd(P ) = 1, hence f0(P ) − f1(P ) + · · · + (−1)dfd(P ) = 1, or f0(P ) − f1(P ) + · · · + (−1)d−1fd−1(P ) = 1 − (−1)d. In 3 dimensions, V − E + F = 2. Back to hk,c(P ), note that if degc(v) = k then deg−c(v) = d − k. Because of independence of c, we obtain the Dehn-Sommerville Relations: hk (P ) = hd−k (P ). 11 Cyclic polytopes and the upper bound theorem A cyclic d-polytope with n vertices is defined by n scalars t1, . . . , tn as conv({(ti, t2i , . . . , tdi ) : i = 1, d}). Can use other curves too. All cyclic d-polytopes with n vertices have same structure, denote by C(d, n). The polar C ∗(d, n) := {x ∈ (Rd)∗ : xv ≤ 1, ∀v ∈ C(d, n)} is a simple polytope. Property: C(d, n) has the maximum number of k-facets for any polytope with n vertices. The polar C ∗(d, n) has the maximum number of k-facets for any polytope with n facets (the face lattice). Exact expression � � for fk−1 elaborate, but a simple one is fk−1 = �min{d,k} d − i h (P ). For more interesting details, see [Zie95]. i=0 k−i i 12 Abstract objective functions and the combinatorial structure An abstract objective function assigns a value to every vertex of a simple polytope P , s.t. every non-empty face F of P has a unique local maximum vertex. AOFs are gen. of linear objective functions. Most results here apply. The combinatorial structure of a polytope is all the information on facet inclusion, e.g. all vertices, all edges and the vertices they are composed of, all 3-facets and their composition, etc. Lemma: Given graph G(P ) of simple polytope P , connected subgraph H = (V (H), E(H)) with k vertices defines a k-face if and only if ∃ AOF s.t. all vertices in V (H) come before all vertices in V (G(P )) \ V (H). Property: The combinatorial structure of any simple polytope is determined by its graph. 13 Main result in this talk—context In the simplex algorithm we often make choices on which vertex to move to next. Criteria for choosing the next vertex are called pivot rules. In the early days, “believed” simple rules guarantee a polynomial number of vertices in path. Klee and Minty [KM72] have shown exponential behaviour. After that, not known even if LP can be solved in polynomial time at all, until [Kha79]. But still, • Finding a pivot rule (deterministic or randomized) that would yield a polynomial number of vertex changes—open since simplex introduced. For some f (n), exponential: f (n) ∈ Ω(k n), k > 1. Polynomial: f (n) ∈ O(nk ) for some fixed k ≥ 1. Subexponential f (n) �∈ O(nk ) for any fixed k ≥ 1 and f (n) �∈ Ω(k n) for any fixed k > 1. 14 Main result in this talk Shortly before a different technique in [SW92], shorty aftewards a subexponential analysis for it in [MSW96]. • The first randomized pivot rule that yields subexponential expected path length (presented from [Kal92a, Kal97]). Expectation over internal random choices of algorithm; applicable to all LP instances. Immediate application to diameter of polytopes and the Hirsch conjecture (more about diameters and the Hirsch conjecture later). 15 Algorithm 1 Simplest randomization (Dantzig, others): next vertex random with equal prob. among neighboring cost-increasing vertices. Hard to analyize in general; Gärtner, Henk and Ziegler show quadratic lower bounds on Klee-Minty cubes. Reminder: P Given P = {x ∈ Rd : Ax ≤ b}, so in LP terms: d = # of variables, n = # of constraints. Also given c ∈ Rd. A1-1 (parameter r, start vertex v): 1. 2. 3. 4. Find vertices on r facets F1, F2, . . . , Fr s.t. ∀Fi, cv < max{cx : x ∈ Fi}. Choose a facet Fk at random from F1, F2, . . . , Fr with equal probability. Solve max{cx : x ∈ Fk } recursively. Let the optimum vertex be w. Finish solving the problem from w recursively. How is this a simplex algorithm? 16 A1-1: Implementation of step 1 For step 1, easy to find the first d facets. k := d, z := v and proceed as follows: For the rest r − d facets, let 1. Solve an LP from z with only the k facets recursively. Let result be z. 2. If z feasible for original problem, optimum found, A1-1 terminates. 3. Otherwise, first edge E on path that leaves P gives new facet F . Let z be the point in E ∩ F . If r facets, stop; otherwise go to step 1. Up to now we are tracing a path along the vertices of the original problem. 17 A1-1: Implementation of steps 2 and 3 First, note that when solving max{cx : x ∈ Fk }, tracing a path along vertices of Fk . This is also a path along vertices of P , since we are working with Fk in its dimension. If k = r or k = r − 1, then last vertex ∈ Fk , can continue our path in step 3. But if k < r − 1, then backtracking from the last vertex found when discovered. Not “honest” simplex. Easy to fix. Since facet Fk is chosen uniformly among facets F1, . . . , Fr , this can be done by choosing uniformly among Fi1 , . . . , Fir , where i1, . . . , ir order in which facets encountered by step 1. So, generate random k before step 1, and stop once reached k-th facet (Kalai also offers another workaround). 18 A1-1: Implementation of step 4 A facet F is active w.r.t. a vertex v if ∃w ∈ vert(F ) s.t. cw > cv. Apply algorithm recursively from w using only those facets which are active. At most n − 1 such facets (Fk from step 3 cannot be active). Complexity analysis Let f1(P, c) := �E[# of pivots when solving max{cx : x ∈ P } by �A1]. Let f1(d, n) := max f1({x ∈ Rd : Ax ≤ b}, c) : A ∈ Rn×d, c ∈ Rd, b ∈ Rn . First part of analysis: probabilistic reasoning to obtain a recurrence relation on f1(d, n). Second part: solving the recurrence (using generating functions). 19 A1-1: Analysis of step 1 It takes f1(d, i)�to solve an LP in d variables with i facets using A1-1. So step 1 r takes at most i=1 f1(d, i). In step 2, note that there is at least one vertex in the path for each random number generated. In step 3, the expected complexity is f1(d − 1, n − 1). After step 3, we only need to consider the active facets w.r.t. w. How many? Assume facets F1, . . . , Fr are ordered according to their top vertex. Then selecting facet i ⇒ at most n − i − 1 active facets w.r.t. w. So with probability 1 r we will have n − i − 1 active facets, for i = 1, 2, . . . , r. �r �n−1 �l Rec.: f1(d, n) = 2 i=d f1(d, i) + f1(d − 1, n − 1) + 1r n−r−1 i=d f1(d, n − i). 20 (The real) Algorithm 1 Before analyzing the recurrence, we improve A1-1. A1 (parameter c > 21 , start vertex v): 1. 2. 3. 4. Starting from v, find vertices on r active facets F1, F2, . . . , Fr . Choose a facet Fk at random from F1, F2, . . . , Fr with equal probability. Solve max{cx : x ∈ Fk } recursively. Let the optimum vertex be w. Let l := |{F : F active w.r.t. w}|. If l > (1 − c)n then let v := w and go to step 1; otherwise, finish solving recursively from w. �n � Let r := max 2 , d . What is probability of not returning to step 1? If r = n easily geometric with ratio = P ( no return) = P (l < (1 − c)n) = 1 − c. In general, analysis more complicated. 21 A1: the recurrence r r � 2 � 1 1 f1(d, n − i)i. f1(d, n) ≤ f1(d, i) + f (d − 1, n − 1) + 1−c 1−c (1 − c)n i=d i=�cn� (1) � � d + logb n 1 Taking b = 1−c , we get a bound of f1(d, n) ≤ bd(6n)logb n . logb n Taking c = 1 − √1d , we obtain √ 16 d f1(d, n) ≤ n . (2) 22 Algorithm 2 Delete step 4, repeat steps 1–3 until step 1 detects an optimal vertex. Equivalent to setting c := 1 in 1. A2 (start vertex v): 1. Starting from v, find vertices on r active facets F1, F2, . . . , Fr . If unable to find r distinct active facets ⇒ opt. vertex found. 2. Choose a facet Fk at random from F1, F2, . . . , Fr with equal probability. 3. Solve max{cx : x ∈ Fk } recursively. Let the optimum vertex be w. 4. Delete inactive facets, set v := w, and go to step 1. Recurrence is: f2(d, n) ≤ f2(d − 1, n − 1) + n/2 � i=d n/2 2� g(d, i) + g(d, n − i). n i=1 (3) 23 A2: the bounds √ C Kd By solving the recurrences, we get f2(d, Kd) ≤ 2 � C , and f2(d, n) ≤ n d log d . When co-dimension (m := n − d) is small the following bound is very useful: √ 2C m log d. Next: the interesting A3. 24 Algorithm 3 A3 (start vertex v): 1. From d facets containing v, select a facet F0 at random, with equal probability. 2. Apply A3 to F0 recursively, and let w be the optimum. 3. Set v := w and go to step 1. Simple! This algorithm is the dual of the algorithm discovered by Sharir and Welzl [SW92] (more about this later). For now, note that in a simple polytope, there can be at most 1 non-active facet adjacent to any vertex v, unless v is optimal. 25 A3: the recursion First, if all facets active, with probability facets at step 3, for i = 1, . . . , d. 1 d the chosen facet yields n − i active Second, if one facet inactive, with probability facets at step 3, for i = 1, . . . , d − 1. 1 d−1 the chosen facet yields n − i Second alternative is worse, so we factor it in and obtain recursion f3(d, n) ≤ �d−1 1 f (d − 1, n − 1) + d−1 i=1 f (d, n − i). This yields bound f3(d, n) ≤ e √ eC d log n, better, like A2. C √ n log d . A4, which we do not present now, gives 26 Subexponential behaviour 2500 9 2·10 2000 9 1.5·10 1500 9 1·10 1000 8 5·10 500 2 4 6 8 10 12 14 200 400 600 800 1000 1200 d 3 Figure 2: Asymptotic behaviour of the exponential 2 , the polynomial d and the √ subexponential 2 d. 27 The duality of A4 and the Sharir-Welzl algorithm Following [Gol95], we show what the Sharir-Welzl algorithm (Algorithm B) [MSW96] does to the polytope of the dual LP. Reminder: algorithm B was called BasisLP in the second part of Session 3. Unlike before, we’ll use lots of traditional LP terminology. Let H be a set of constraints, and B a set of constraints that define a basis. B 1. 2. 3. 4. (set of constraints H, basis C with C ⊆ H): Begin at C. Choose random constraint h ∈ H \ C Solve LP recursively with constraints H \ {h}, from C. Let result be B. If B violates h, then form new basis C � := basis(B, h); otherwise optimum found. 5. Let C := C � and go to step 1. 28 The dual problems Primal (we run B on this problem): max{cx : Ax ≤ b}. (4) Dual (we see what B does to this problem): min{yb : y ≥ 0, yA = c}. (5) We will imagine the dual problem in the yA = c space, so only the inequality constraints y ≥ 0 define facets; yA = c is simply an affine transformation of the space. 29 The correspondence, up to step 3 C := initial feasible basis � C := initial feasible vertex. (Not true.) Choosing random h ∈ H \ C � choosing random facet yh ≥ 0 that contains C: • We know, by complementary slackness, that un-tight constraints in the primal correspond to 0-level variable components in the dual. • Only the yi ≥ 0 constraints define facets in the dual polytope. • Active constraint at a vertex C defines a facet that constains C. Solve recursively the LP with constrains H \h starting from C � solve recursively LP on facet yh = 0 starting from C. • If we remove a constraint in the primal, this is the same as requiring yh = 0 in the dual. 30 The correspondence, step 4 In both cases we obtain a new basis (� vertex) B. If B violates h � B not optimal: infeasible primal solutions correspond to suboptimal dual solutions. C � := basis(B, h) is a pivot operation � C � := move away along an unique edge from yh = 0. But, there is no “move along an edge” in A3! Slight adjustment to A3, in order to achieve perfect duality: in step 1, pick a random facet among d active facets. After we found optimum w on facet F0, this facet is not active (since w optimum on it). Moreover, this facet is defined by d − 1 of the edges at w (in a simple polytope, every d − 1 edges at a vertex define a facet, and conversely). 31 The correspondence, steps 4 and 5 Hence only one remaining edge can be cost-improving ⇒ any simplex algorithm will take it. So, our algorithm takes it when it tries to find the d-th facet. But this implies that the choice of d facets available to A3 is exactly the same as the choice of d facets available after moving along the unique edge. In steps 5 and 1 this yields the same choice of un-tight constraints in the primal! So, a variant of the Sharir-Welzl algorithm B when followed on the dual polytope is exactly the same as the (slightly modified) Kalai algorithm A3. 32 The Hirsch conjecture The diameter δ(G(P )) of a polytope is the diameter of its graph, i.e. the longest � )) the longest shortest path between any pair (v, w) of vertices. Denote by δ(G(P shortest path in the cost-function directed graph of P . Let Δ(d, n) := max{δ(G(P )) : P is a d-polytope with n facets }. Let H(d, n) := max{δ(G(P )) : P is a d-polytope with n facets, c is any cost function }. Clearly, the simplex algorithm cannot guarantee a better performance on P than δ(G(P )). Moreover, Δ(d, n) ≤ H(d, n). Conjecture (Hirsch, [Dan63]): Δ(P ) ≤ n − d. False for unbounded polyhedra [KW67]. Lower bound of Δ(d, n) ≥ n− d+ �d/5�. Still best lower bound! 33 Status of Hirsch conjecture for polytopes Still open! Exponential bound Δ(d, n) ≤ n2d−3 [Larman, 1970]. Until recently (w.r.t. 1992) no sub-exponential bound known. Bounds of n2 log d+3 and nlog d+1 in [Kal92b, KK92] respectively. How randomized pivot rules affect the Hirsch conjecture? A randomized simplex algorithm gives only hope that a deterministic algorithm with the same complexity may be devised. But, because E[...] is over choices, at least one of these choices (even if we don’t know it), yields a path of length less that E[...]. 34 A friendly oracle So all our bounds on the number of simplex pivots, immediately apply to H(d, n) (since simplex takes only monotone paths) and hence to Δ(d, n). In algorithms A1–A4 we spend at most O(d2n) for each pivot, and generate at most 1 random number per pivot. What if we allow much more time per pivot? Result will still apply to the Hirsch conjecture. Do not want algorithms such as “construct the graph, find the shortest path, then parse it”, since analysis is equivalent to Hirsch conjecture. But, can still make use of a more powerful oracle that makes choices at each pivot step. Works from within Algorithm 4. 35 Algorithm 4 A4 (start vertex v): 1. From the active facets w.r.t v, select a facet F0 at random, with equal probability. 2. Find vertices recursively until reached F0, or until optimum found. 3. Solve recursively on F0. Let result be v. 4. Go to step 1. C As mentioned, bound of e √ d log n . Now instead of selecting F0 at random, order all facets in increasing order F1, . . . , Fn of max{cx : x ∈ Fi}. Select F0 s.t. max{x : x ∈ F0} above the median (i > �n/2� in the ordering). Let f4(d, n) be the number of steps using the oracle. 36 A4: the recursion At most f4(d, n/2) pivots for step 2. Pivots from step 4 (counting everything that happens) is again f4(d, n/2). Clearly, step 3 takes f4(d − 1, n − 1). Hence: f4(d, n) ≤ 2f4(d, �n/2�) + f4(d − 1, n − 1) + 1. (6) To solve this, let φ(d, t) := 2tf4(d, 2t). Then, from (6), we obtain φ(d, t) ≤ φ(d − 1, t) + φ(d, t − 1). By simple�combinatorial reasoning (counting all�paths to the bottom), this yields � � d+t log n + d φ(d, t) ≤ . So f4(d, n) ≤ n . d log n � � a+b Finally, by combinatorics ≤ ab (or ba), we obtain f4(d, n) ≤ nlog d+1. a 37