Information Theory
- [1] arXiv:2405.09010 [pdf, ps, html, other]
-
Title: On Low Field Size Constructions of Access-Optimal Convertible CodesComments: This is an extended version of an IEEE ISIT 2024 paper with the same titleSubjects: Information Theory (cs.IT)
Most large-scale storage systems employ erasure coding to provide resilience against disk failures. Recent work has shown that tuning this redundancy to changes in disk failure rates leads to substantial storage savings. This process requires code conversion, wherein data encoded using an $[n^{I\mskip-2mu},k^{I\mskip-2mu}]$ initial code has to be transformed into data encoded using an $[n^{F\mskip-2mu},k^{F\mskip-2mu}]$ final code, a resource-intensive operation. Convertible codes are a class of codes that enable efficient code conversion while maintaining other desirable properties. In this paper, we focus on the access cost of conversion (total number of code symbols accessed in the conversion process) and on an important subclass of conversions known as the merge regime (combining multiple initial codewords into a single final codeword).
In this setting, explicit constructions are known for systematic access-optimal Maximum Distance Separable (MDS) convertible codes for all parameters in the merge regime. However, the existing construction for a key subset of these parameters, which makes use of Vandermonde parity matrices, requires a large field size making it unsuitable for practical applications. In this paper, we provide (1) sharper bounds on the minimum field size requirement for such codes, and (2) explicit constructions for low field sizes for several parameter ranges. In doing so, we provide a proof of super-regularity of specially designed classes of Vandermonde matrices that could be of independent interest. - [2] arXiv:2405.09200 [pdf, ps, html, other]
-
Title: Performance Analysis of RIS-aided MISO Systems with EMI and Channel AgingSubjects: Information Theory (cs.IT)
In this paper, we investigate a reconfigurable intelligent surface (RIS)-aided multiple-input single-output (MISO) system in the presence of electromagnetic interference (EMI) and channel aging with a Rician fading channel model between the base station (BS) and user equipment (UE). Specifically, we derive the closed-form expression for downlink spectral efficiency (SE) with maximum ratio transmission (MRT) precoding. The Monte-Carlo simulation supports the theoretical results, demonstrating that amplifying the weight of the line-of-sight (LoS) component in Rician fading channels can boost SE, while EMI has a detrimental impact. Furthermore, continuously increasing the number of RIS elements is not an optimal choice when EMI exists. Nonetheless, RIS can be deployed to compensate for SE degradation caused by channel aging effects. Finally, enlarging the RIS elements size can significantly improve system performance.
- [3] arXiv:2405.09207 [pdf, ps, html, other]
-
Title: An Exact Theory of Causal Emergence for Linear Stochastic Iteration SystemsSubjects: Information Theory (cs.IT); Systems and Control (eess.SY)
After coarse-graining a complex system, the dynamics of its macro-state may exhibit more pronounced causal effects than those of its micro-state. This phenomenon, known as causal emergence, is quantified by the indicator of effective information. However, two challenges confront this theory: the absence of well-developed frameworks in continuous stochastic dynamical systems and the reliance on coarse-graining methodologies. In this study, we introduce an exact theoretic framework for causal emergence within linear stochastic iteration systems featuring continuous state spaces and Gaussian noise. Building upon this foundation, we derive an analytical expression for effective information across general dynamics and identify optimal linear coarse-graining strategies that maximize the degree of causal emergence when the dimension averaged uncertainty eliminated by coarse-graining has an upper bound. Our investigation reveals that the maximal causal emergence and the optimal coarse-graining methods are primarily determined by the principal eigenvalues and eigenvectors of the dynamic system's parameter matrix, with the latter not being unique. To validate our propositions, we apply our analytical models to three simplified physical systems, comparing the outcomes with numerical simulations, and consistently achieve congruent results.
- [4] arXiv:2405.09309 [pdf, ps, html, other]
-
Title: Identification via Binary Uniform Permutation ChannelComments: 9 pages. Extended version of submission to ITW 2024Subjects: Information Theory (cs.IT)
We study message identification over the binary uniform permutation channels. For DMCs, the number of identifiable messages grows doubly exponentially. Identification capacity, the maximum second-order exponent, is known to be the same as the Shannon capacity of a DMC. We consider a binary uniform permutation channel where the transmitted vector is permuted by a permutation chosen uniformly at random. Permutation channels support reliable communication of only polynomially many messages. While this implies a zero second-order identification rate, we prove a soft converse result showing that even non-zero first-order identification rates are not achievable with a power-law decay of error probability for identification over binary uniform permutation channels. To prove the converse, we use a sequence of steps to construct a new identification code with a simpler structure and then use a lower bound on the normalized maximum pairwise intersection of a set system on {0, . . . , n}. We provide generalizations for arbitrary alphabet size.
- [5] arXiv:2405.09336 [pdf, ps, html, other]
-
Title: Analytical Characterization of the Operational Diversity Order in Fading ChannelsSubjects: Information Theory (cs.IT); Signal Processing (eess.SP)
We introduce and characterize the operational diversity order (ODO) in fading channels, as a proxy to the classical notion of diversity order at any arbitrary operational signal-to-noise ratio (SNR). Thanks to this definition, relevant insights are brought up in a number of cases: (i) We quantify that in line-of-sight scenarios an increased diversity order is attainable compared to that achieved asymptotically; (ii) this effect is attenuated, but still visible, in the presence of an additional dominant specular component; (iii) we confirm that the decay slope in Rayleigh product channels increases very slowly and never fully achieves unitary slope for finite values of SNR.
- [6] arXiv:2405.09425 [pdf, ps, html, other]
-
Title: Robust Covariance-Based Activity Detection for Massive AccessComments: 5 pages, 11 figures. Asilomar SSC 2023 ConferenceSubjects: Information Theory (cs.IT); Signal Processing (eess.SP)
The wireless channel is undergoing continuous changes, and the block-fading assumption, despite its popularity in theoretical contexts, never holds true in practical scenarios. This discrepancy is particularly critical for user activity detection in grant-free random access, where joint processing across multiple resource blocks is usually undesirable. In this paper, we propose employing a low-dimensional approximation of the channel to capture variations over time and frequency and robustify activity detection algorithms. This approximation entails projecting channel fading vectors onto their principal directions to minimize the approximation order. Through numerical examples, we demonstrate a substantial performance improvement achieved by the resulting activity detection algorithm.
- [7] arXiv:2405.09443 [pdf, ps, html, other]
-
Title: Low-Complexity Joint Azimuth-Range-Velocity Estimation for Integrated Sensing and Communication with OFDM WaveformComments: 16 pages, 12 figures, submitted to IEEE journalSubjects: Information Theory (cs.IT); Signal Processing (eess.SP)
Integrated sensing and communication (ISAC) is a main application scenario of the sixth-generation mobile communication systems. Due to the fast-growing number of antennas and subcarriers in cellular systems, the computational complexity of joint azimuth-range-velocity estimation (JARVE) in ISAC systems is extremely high. This paper studies the JARVE problem for a monostatic ISAC system with orthogonal frequency division multiplexing (OFDM) waveform, in which a base station receives the echos of its transmitted cellular OFDM signals to sense multiple targets. The Cramer-Rao bounds are first derived for JARVE. A low-complexity algorithm is further designed for super-resolution JARVE, which utilizes the proposed iterative subspace update scheme and Levenberg-Marquardt optimization method to replace the exhaustive search of spatial spectrum in multiple-signal-classification (MUSIC) algorithm. Finally, with the practical parameters of 5G New Radio, simulation results verify that the proposed algorithm can reduce the computational complexity by three orders of magnitude and two orders of magnitude compared to the existing three-dimensional MUSIC algorithm and estimation-of-signal-parameters-using-rotational-invariance-techniques (ESPRIT) algorithm, respectively, and also improve the estimation performance.
- [8] arXiv:2405.09497 [pdf, ps, html, other]
-
Title: Towards the limits: Sensing Capability Measurement for ISAC Through Channel EncoderSubjects: Information Theory (cs.IT); Networking and Internet Architecture (cs.NI); Signal Processing (eess.SP)
Integrated Sensing and Communication (ISAC) is gradually becoming a reality due to the significant increase in frequency and bandwidth of next-generation wireless communication technologies. Therefore it becomes crucial to evaluate the communication and sensing performance using appropriate channel models to address resource competition from each other. Existing work only models the sensing capability based on the mutual information between the channel response and the received signal, and its theoretical resolution is difficult to support the high-precision requirements of ISAC for sensing tasks, and may even affect its communication optimal.
In this paper, we propose a sensing channel encoder model to measure the sensing capacity with higher resolution by discrete task mutual information. For the first time, derive upper and lower bounds on the sensing accuracy for a given channel. This model not only provides the possibility of optimizing the ISAC systems at a finer granularity and balancing communication and sensing resources, but also provides theoretical explanations for classical intuitive feelings (like more modalities more accuracy) in wireless sensing. Furthermore, we validate the effectiveness of the proposed channel model through real-case studies, including person identification, displacement detection, direction estimation, and device recognition. The evaluation results indicate a Pearson correlation coefficient exceeding 0.9 between our task mutual information and conventional experimental metrics (e.g., accuracy). - [9] arXiv:2405.09534 [pdf, ps, html, other]
-
Title: Learning-Based Compress-and-Forward Schemes for the Relay ChannelComments: journal submission under review. arXiv admin note: substantial text overlap with arXiv:2404.14594Subjects: Information Theory (cs.IT)
The relay channel, consisting of a source-destination pair along with a relay, is a fundamental component of cooperative communications. While the capacity of a general relay channel remains unknown, various relaying strategies, including compress-and-forward (CF), have been proposed. In CF, the relay forwards a quantized version of its received signal to the destination. Given the correlated signals at the relay and destination, distributed compression techniques, such as Wyner--Ziv coding, can be harnessed to utilize the relay-to-destination link more efficiently. Leveraging recent advances in neural network-based distributed compression, we revisit the relay channel problem and integrate a learned task-aware Wyner--Ziv compressor into a primitive relay channel with a finite-capacity out-of-band relay-to-destination link. The resulting neural CF scheme demonstrates that our compressor recovers binning of the quantized indices at the relay, mimicking the optimal asymptotic CF strategy, although no structure exploiting the knowledge of source statistics was imposed into the design. The proposed neural CF, employing finite order modulation, operates closely to the rate achievable in a primitive relay channel with a Gaussian codebook. We showcase the advantages of exploiting the correlated destination signal for relay compression through various neural CF architectures that involve end-to-end training of the compressor and the demodulator components. Our learned task-oriented compressors provide the first proof-of-concept work toward interpretable and practical neural CF relaying schemes.
New submissions for Thursday, 16 May 2024 (showing 9 of 9 entries )
- [10] arXiv:2405.09079 (cross-list from eess.SP) [pdf, ps, html, other]
-
Title: Integrated Monostatic Sensing and Full-Duplex Multiuser Communication for mmWave SystemsComments: 13 pages, 7 figuresSubjects: Signal Processing (eess.SP); Information Theory (cs.IT)
In this paper, we propose a hybrid precoding/combining framework for communication-centric integrated sensing and full-duplex (FD) communication operating at mmWave bands. The designed precoders and combiners enable multiuser (MU) FD communication while simultaneously supporting monostatic sensing in a frequency-selective setting. The joint design of precoders and combiners involves the mitigation of self-interference (SI) caused by simultaneous transmission and reception at the FD base station (BS). Additionally, MU interference needs to be handled by the precoder/combiner design. The resulting optimization problem involves non-convex constraints since hybrid analog/digital architectures utilize networks of phase shifters. To solve the proposed problem, we separate the optimization of each precoder/combiner, and design each one of them while fixing the others. The precoders at the FD BS are designed by reformulating the communication and sensing constraints as signal-to-leakage-plus-noise ratio (SLNR) maximization problems that consider SI and MU interference as leakage. Furthermore, we design the frequency-flat analog combiner such that the residual SI at the FD BS is minimized under communication and sensing gain constraints. Finally, we design an interference-aware digital combining stage that separates MU signals and target reflections. The communication performance and sensing results show that the proposed framework efficiently supports both functionalities simultaneously.
- [11] arXiv:2405.09254 (cross-list from math.CO) [pdf, ps, html, other]
-
Title: Eigenvalue bounds and alternating rank-metric codesSubjects: Combinatorics (math.CO); Information Theory (cs.IT)
In this note we apply a spectral method to the graph of alternating bilinear forms. In this way, we obtain upper bounds on the size of an alternating rank-metric code for given values of the minimum rank distance. We computationally compare our results with Delsarte's linear programming bound, observing that they give the same value. For small values of the minimum rank distance, we are able to establish the equivalence of the two methods. The problem remains open for larger values.
- [12] arXiv:2405.09283 (cross-list from eess.SP) [pdf, ps, html, other]
-
Title: Bounds and Approximations for the Distribution of a Sum of Lognormal Random VariablesSubjects: Signal Processing (eess.SP); Information Theory (cs.IT)
A sum of lognormal random variables (RVs) appears in many problems of science and engineering. For example, it is invloved in computing the distribution of recevied signal and interference powers for radio channels subject to lognormal shadow fading. Its distribution has no closed-from expression and it is typically characterized by approximations, asymptotes or bounds. We give a novel upper bound on the cumulative distribution function (CDF) of a sum of $N$ lognormal RVs. The bound is derived from the tangential mean-arithmetic mean inequality. By using the tangential mean, our method replaces the sum of $N$ lognormal RVs with a product of $N$ shifted lognormal RVs. It is shown that the bound can be made arbitrarily close to the desired CDF, and thus it becomes more accurate than any other bound or approximation, as the shift approaches infinity. The bound is computed by numerical integration, for which we introduce the Mellin transform, which is applicable to products of RVs. At the left tail of the CDF, the bound can be expressed by a single Q-function. Moreover, we derive simple new approximations to the CDF, expressed as a product $N$ Q-functions, which are more accurate than the previous method of Farley.
- [13] arXiv:2405.09304 (cross-list from cs.DM) [pdf, ps, html, other]
-
Title: Kolmogorov complexity as a combinatorial toolComments: Prepared as an special session invited talk at CiE 2024Subjects: Discrete Mathematics (cs.DM); Information Theory (cs.IT); Combinatorics (math.CO)
Kolmogorov complexity is often used as a convenient language for counting and/or probabilistic existence proofs. However, there are some applications where Kolmogorov complexity is used in a more subtle way. We provide one (somehow) surprising example where an existence of a winning strategy in a natural combinatorial game is proven (and no direct proof is known).
- [14] arXiv:2405.09455 (cross-list from stat.CO) [pdf, ps, html, other]
-
Title: Efficient pooling designs and screening performance in group testing for two type defectivesSubjects: Computation (stat.CO); Information Theory (cs.IT)
Group testing is utilized in the case when we want to find a few defectives among large amount of items. Testing n items one by one requires n tests, but if the ratio of defectives is small, group testing is an efficient way to reduce the number of tests. Many research have been developed for group testing for a single type of defectives. In this paper, we consider the case where two types of defective A and B exist. For two types of defectives, we develop a belief propagation algorithm to compute marginal posterior probability of defectives. Furthermore, we construct several kinds of collections of pools in order to test for A and B. And by utilizing our belief propagation algorithm, we evaluate the performance of group testing by conducting simulations.
- [15] arXiv:2405.09514 (cross-list from eess.SP) [pdf, ps, html, other]
-
Title: Tackling Distribution Shifts in Task-Oriented Communication with Information BottleneckComments: 13 pages, 8 figures, submitted to IEEE for potential publicationSubjects: Signal Processing (eess.SP); Information Theory (cs.IT); Machine Learning (cs.LG)
Task-oriented communication aims to extract and transmit task-relevant information to significantly reduce the communication overhead and transmission latency. However, the unpredictable distribution shifts between training and test data, including domain shift and semantic shift, can dramatically undermine the system performance. In order to tackle these challenges, it is crucial to ensure that the encoded features can generalize to domain-shifted data and detect semanticshifted data, while remaining compact for transmission. In this paper, we propose a novel approach based on the information bottleneck (IB) principle and invariant risk minimization (IRM) framework. The proposed method aims to extract compact and informative features that possess high capability for effective domain-shift generalization and accurate semantic-shift detection without any knowledge of the test data during training. Specifically, we propose an invariant feature encoding approach based on the IB principle and IRM framework for domainshift generalization, which aims to find the causal relationship between the input data and task result by minimizing the complexity and domain dependence of the encoded feature. Furthermore, we enhance the task-oriented communication with the label-dependent feature encoding approach for semanticshift detection which achieves joint gains in IB optimization and detection performance. To avoid the intractable computation of the IB-based objective, we leverage variational approximation to derive a tractable upper bound for optimization. Extensive simulation results on image classification tasks demonstrate that the proposed scheme outperforms state-of-the-art approaches and achieves a better rate-distortion tradeoff.
- [16] arXiv:2405.09523 (cross-list from math.ST) [pdf, ps, html, other]
-
Title: On Semi-supervised Estimation of Discrete Distributions under f-divergencesComments: Full version. Presented in ISIT-24. arXiv admin note: text overlap with arXiv:2305.07955Subjects: Statistics Theory (math.ST); Information Theory (cs.IT)
We study the problem of estimating the joint probability mass function (pmf) over two random variables. In particular, the estimation is based on the observation of $m$ samples containing both variables and $n$ samples missing one fixed variable. We adopt the minimax framework with $l^p_p$ loss functions. Recent work established that univariate minimax estimator combinations achieve minimax risk with the optimal first-order constant for $p \ge 2$ in the regime $m = o(n)$, questions remained for $p \le 2$ and various $f$-divergences. In our study, we affirm that these composite estimators are indeed minimax optimal for $l^p_p$ loss functions, specifically for the range $1 \le p \le 2$, including the critical $l_1$ loss. Additionally, we ascertain their optimality for a suite of $f$-divergences, such as KL, $\chi^2$, Squared Hellinger, and Le Cam divergences.
- [17] arXiv:2405.09525 (cross-list from quant-ph) [pdf, ps, html, other]
-
Title: Improved classical shadows from local symmetries in the Schur basisSubjects: Quantum Physics (quant-ph); Data Structures and Algorithms (cs.DS); Information Theory (cs.IT); Machine Learning (cs.LG)
We study the sample complexity of the classical shadows task: what is the fewest number of copies of an unknown state you need to measure to predict expected values with respect to some class of observables? Large joint measurements are likely required in order to minimize sample complexity, but previous joint measurement protocols only work when the unknown state is pure. We present the first joint measurement protocol for classical shadows whose sample complexity scales with the rank of the unknown state. In particular we prove $\mathcal O(\sqrt{rB}/\epsilon^2)$ samples suffice, where $r$ is the rank of the state, $B$ is a bound on the squared Frobenius norm of the observables, and $\epsilon$ is the target accuracy. In the low-rank regime, this is a nearly quadratic advantage over traditional approaches that use single-copy measurements.
We present several intermediate results that may be of independent interest: a solution to a new formulation of classical shadows that captures functions of non-identical input states; a generalization of a ``nice'' Schur basis used for optimal qubit purification and quantum majority vote; and a measurement strategy that allows us to use local symmetries in the Schur basis to avoid intractable Weingarten calculations in the analysis.
Cross submissions for Thursday, 16 May 2024 (showing 8 of 8 entries )
- [18] arXiv:2308.02324 (replaced) [pdf, ps, html, other]
-
Title: Robust mmWave/sub-THz multi-connectivity using minimal coordination and coarse synchronizationComments: Major revision: added ray-tracing simulation to validate the theoretical analysis, and refactored the presentation to avoid misleading connections with the canonical cell-free massive MIMO literatureSubjects: Information Theory (cs.IT); Signal Processing (eess.SP)
This study investigates simpler alternatives to coherent joint transmission for supporting robust connectivity against signal blockage in mmWave/sub-THz access networks. By taking an information-theoretic viewpoint, we demonstrate analytically that with a careful design, full macrodiversity gains and significant SNR gains can be achieved through canonical receivers and minimal coordination and synchronization requirements at the infrastructure side. Our proposed scheme extends non-coherent joint transmission by employing a special form of diversity to counteract artificially induced deep fades that would otherwise make this technique often compare unfavorably against standard transmitter selection schemes. Additionally, the inclusion of an Alamouti-like space-time coding layer is shown to recover a significant fraction of the optimal performance. Our conclusions are based on an insightful multi-point intermittent block fading channel model that enables rigorous ergodic and outage rate analysis, while also considering timing offsets due to imperfect delay compensation. Although simplified, our approach captures the essential features of modern mmWave/sub-THz communications, thereby providing practical design guidelines for realistic systems.
- [19] arXiv:2401.16288 (replaced) [pdf, ps, html, other]
-
Title: Upper bounds on the rate of linear $q$-ary $k$-hash codesSubjects: Information Theory (cs.IT); Combinatorics (math.CO)
This paper presents new upper bounds on the rate of linear $k$-hash codes in $\mathbb{F}_q^n$, $q\geq k$, that is, codes with the property that any $k$ distinct codewords are all simultaneously distinct in at least one coordinate.
- [20] arXiv:2402.11533 (replaced) [pdf, ps, html, other]
-
Title: Randomness-Efficient Constructions of Capacity-Achieving List-Decodable CodesSubjects: Information Theory (cs.IT)
We wish to generate list-decodable codes over small alphabets using as little randomness as possible. Specifically, we hope to generate codes achieving what we term the Elias bound, which means that they are $(\rho,L)$-list-decodable with rate $R \geq 1-h(\rho)-O(1/L)$. A long line of work shows that uniformly random linear codes (RLCs) achieve the Elias bound: hence, we know $O(n^2)$ random bits suffice. Prior works demonstrate that just $O(Ln)$ random bits suffice, via puncturing of low-bias codes. These recent constructions are combinatorial.
We provide two new constructions, which are algebraic. Compared to prior works, our constructions are simpler and more direct. Furthermore, our codes are designed in such a way that their duals are also quite easy to analyze. Our first construction -- which can be seen as a generalization of the Wozencraft ensemble -- achieves the Elias bound and consumes $Ln$ random bits. Additionally, its dual code achieves the GV-bound with high probability, and both the primal and dual admit quasilinear-time encoding algorithms. The second construction consumes $2nL$ random bits and yields a code where both it and its dual achieve the Elias bound. As we discuss, properties of a dual code are often crucial for applications in cryptography.
In all of the above cases -- including the prior works achieving randomness complexity $O(Ln)$ -- the codes are designed to "approximate" RLCs. Namely, for a given locality parameter $L$ we construct codes achieving the same $L$-local properties as RLCs. This allows one to appeal to known list-decodability results for RLCs and thereby conclude that the code approximating an RLC also achieves the Elias bound. As a final contribution, we indicate that such a proof strategy is inherently unable to generate list-decodable codes of rate $R$ over $\mathbb F_q$ with less than $L(1-R)n\log_2(q)$ bits of randomness. - [21] arXiv:2403.10656 (replaced) [pdf, ps, html, other]
-
Title: Properties of the Strong Data Processing Constant for R\'enyi DivergenceComments: 6 pages, 1 figureSubjects: Information Theory (cs.IT)
Strong data processing inequalities (SDPI) are an important object of study in Information Theory and have been well studied for $f$-divergences. Universal upper and lower bounds have been provided along with several applications, connecting them to impossibility (converse) results, concentration of measure, hypercontractivity, and so on. In this paper, we study Rényi divergence and the corresponding SDPI constant whose behavior seems to deviate from that of ordinary $\Phi$-divergences. In particular, one can find examples showing that the universal upper bound relating its SDPI constant to the one of Total Variation does not hold in general. In this work, we prove, however, that the universal lower bound involving the SDPI constant of the Chi-square divergence does indeed hold. Furthermore, we also provide a characterization of the distribution that achieves the supremum when $\alpha$ is equal to $2$ and consequently compute the SDPI constant for Rényi divergence of the general binary channel.
- [22] arXiv:2403.18621 (replaced) [pdf, ps, html, other]
-
Title: Performance Analysis of Integrated Sensing and Communication Networks with Blockage EffectsComments: Submitted to IEEE Transactions on Vehicular TechnologySubjects: Information Theory (cs.IT); Signal Processing (eess.SP)
Communication-sensing integration represents an up-and-coming area of research, enabling wireless networks to simultaneously perform communication and sensing tasks. However, in urban cellular networks, the blockage of buildings results in a complex signal propagation environment, affecting the performance analysis of integrated sensing and communication (ISAC) networks. To overcome this obstacle, this paper constructs a comprehensive framework considering building blockage and employs a distance-correlated blockage model to analyze interference from line of sight (LoS), non-line of sight (NLoS), and target reflection cascading (TRC) links. Using stochastic geometric theory, expressions for signal-to-interference-plus-noise ratio (SINR) and coverage probability for communication and sensing in the presence of blockage are derived, allowing for a comprehensive comparison under the same parameters. The research findings indicate that blockage can positively impact coverage, especially in enhancing communication performance. The analysis also suggests that there exists an optimal base station (BS) density when blockage is of the same order of magnitude as the BS density, maximizing communication or sensing coverage probability.
- [23] arXiv:2405.04144 (replaced) [pdf, ps, html, other]
-
Title: Lossy Compression with Data, Perception, and Classification ConstraintsComments: 23 pages, in part submitted to ITWSubjects: Information Theory (cs.IT)
By extracting task-relevant information while maximally compressing the input, the information bottleneck (IB) principle has provided a guideline for learning effective and robust representations of the target inference. However, extending the idea to the multi-task learning scenario with joint consideration of generative tasks and traditional reconstruction tasks remains unexplored. This paper addresses this gap by reconsidering the lossy compression problem with diverse constraints on data reconstruction, perceptual quality, and classification accuracy. Firstly, we study two ternary relationships, namely, the rate-distortion-classification (RDC) and rate-perception-classification (RPC). For both RDC and RPC functions, we derive the closed-form expressions of the optimal rate for binary and Gaussian sources. These new results complement the IB principle and provide insights into effectively extracting task-oriented information to fulfill diverse objectives. Secondly, unlike prior research demonstrating a tradeoff between classification and perception in signal restoration problems, we prove that such a tradeoff does not exist in the RPC function and reveal that the source noise plays a decisive role in the classification-perception tradeoff. Finally, we implement a deep-learning-based image compression framework, incorporating multiple tasks related to distortion, perception, and classification. The experimental results coincide with the theoretical analysis and verify the effectiveness of our generalized IB in balancing various task objectives.
- [24] arXiv:2202.04294 (replaced) [pdf, ps, html, other]
-
Title: Optimal Clustering with Bandit FeedbackComments: 54 pages, 4 figuresSubjects: Machine Learning (cs.LG); Information Theory (cs.IT); Machine Learning (stat.ML)
This paper considers the problem of online clustering with bandit feedback. A set of arms (or items) can be partitioned into various groups that are unknown. Within each group, the observations associated to each of the arms follow the same distribution with the same mean vector. At each time step, the agent queries or pulls an arm and obtains an independent observation from the distribution it is associated to. Subsequent pulls depend on previous ones as well as the previously obtained samples. The agent's task is to uncover the underlying partition of the arms with the least number of arm pulls and with a probability of error not exceeding a prescribed constant $\delta$. The problem proposed finds numerous applications from clustering of variants of viruses to online market segmentation. We present an instance-dependent information-theoretic lower bound on the expected sample complexity for this task, and design a computationally efficient and asymptotically optimal algorithm, namely Bandit Online Clustering (BOC). The algorithm includes a novel stopping rule for adaptive sequential testing that circumvents the need to exactly solve any NP-hard weighted clustering problem as its subroutines. We show through extensive simulations on synthetic and real-world datasets that BOC's performance matches the lower bound asymptotically, and significantly outperforms a non-adaptive baseline algorithm.
- [25] arXiv:2405.08608 (replaced) [pdf, ps, html, other]
-
Title: On the Paley RIP and Paley graph extractorComments: 10 pages, references are updated, comments are welcomeSubjects: Combinatorics (math.CO); Discrete Mathematics (cs.DM); Information Theory (cs.IT); Number Theory (math.NT)
Constructing explicit RIP matrices is an open problem in compressed sensing theory. In particular, it is quite challenging to construct explicit RIP matrices that break the square-root bottleneck. On the other hand, providing explicit $2$-source extractors is a fundamental problem in theoretical computer science, cryptography and combinatorics. Nowadays, there are only a few known constructions for explicit $2$-source extractors (with negligible errors) that break the half barrier for min-entropy.
In this paper, we establish a new connection between RIP matrices breaking the square-root bottleneck and $2$-source extractors breaking the half barrier for min-entropy. Here we focus on an RIP matrix (called the Paley ETF) and a $2$-source extractor (called the Paley graph extractor), where both are defined from quadratic residues over the finite field of odd prime order $p\equiv 1 \pmod{4}$. As a main result, we prove that if the Paley ETF breaks the square-root bottleneck, then the Paley graph extractor breaks the half barrier for min-entropy as well. Since it is widely believed that the Paley ETF breaks the square-root bottleneck, our result accordingly provides a new affirmative intuition on the conjecture for the Paley graph extractor by Benny Chor and Oded Goldreich.