Eigenmode analysis of dielectric waveguides using nyström-method-based boundary integral equations

Abstract

Optical waveguides are fundamental components of modern photonic systems, including integrated optical circuits, fiber-optic communication systems, sensors, and photonic crystal fiber devices. The accurate computation of their guided and leaky eigenmodes is therefore an essential task in computational photonics. In these problems, the main quantities of interest are the propagation constants, effective refractive indices, and corresponding electromagnetic field distributions supported by a given cross-sectional geometry. Standard domain-based methods such as finite-difference and finite-element techniques are widely used for this purpose; however, they usually require volumetric meshing of the cross section and, for open structures, additional truncation or absorbing-boundary treatments. These requirements may become computationally demanding when high accuracy is needed, especially for waveguides with multiple dielectric interfaces, high refractive-index contrast, or geometrically complex cross sections. This thesis develops a high-order boundary-integral eigenmode solver for dielectric optical waveguides. The formulation starts from the time-harmonic Maxwell equations in longitudinally invariant structures and reduces the modal problem to a coupled system for the longitudinal electric and magnetic field components. The transverse field components are then obtained from these longitudinal components through the modal relations. By enforcing the electromagnetic transmission conditions on material interfaces, the waveguide eigenmode problem is written as a nonlinear eigenvalue problem with respect to the propagation constant. The boundary-integral formulation used in this work is based on a Müller-type combined-field representation. This representation is particularly suitable for dielectric transmission problems because it leads to a well-conditioned system involving boundary traces on the material interfaces, while avoiding the need for volumetric discretization of homogeneous subdomains. The resulting boundary-integral operators are discretized using a high-order Nyström method. Smooth interfaces are represented by periodic parametric curves, and the singular or weakly singular behavior of the integral kernels is treated through corrected quadrature rules. Tangential derivatives appearing in the full-vector interface conditions are evaluated consistently along the boundary parametrization. This discretization strategy provides an accurate and geometrically flexible framework for the analysis of waveguides with arbitrary smooth cross sections. In addition, graded-index fiber profiles are approximated by a multilayer representation, which allows continuously varying refractive-index distributions to be incorporated into the same boundary-integral framework. A central difficulty in boundary-only eigenmode computations is the appearance of spurious minima or nonphysical candidate roots during singular-value-based searches. To address this issue, the thesis employs an interior-augmented eigenvalue search strategy based on a thin QR decomposition. Instead of relying only on the smallest singular value of the boundary system, the method augments the boundary equations with interior field samples and constructs a scale-invariant indicator that distinguishes physical modes from spurious boundary-density oscillations. This produces a more reliable search procedure for both guided modes with real propagation constants and leaky modes with complex effective indices. The proposed method is validated through several numerical examples, including square-core or rectangular-core dielectric waveguides, photonic crystal fiber geometries, and graded-index fiber configurations. The computed propagation constants and effective refractive indices are compared with benchmark results from the literature where available, and the corresponding modal field distributions are reconstructed to verify the physical behavior of the obtained modes. The numerical results demonstrate that the proposed high-order Nyström boundary-integral framework can accurately compute waveguide eigenmodes while using only interface-based discretization. The thesis therefore provides a detailed formulation, implementation, and validation of a boundary-integral approach for the eigenmode analysis of dielectric waveguides, with particular emphasis on high-order discretization and spurious-free spectral search.