本书重点介绍经典信息论的基本理论，并力图将信息论的基本理论和工程应用的编码理论联系起来，介绍一些关于这些理论的实际应用。全书分为7章，内容包括信息度量的基本理论、无失真信源编码、限失真信源编码、信道编码及其应用等。

本书注重基本概念，并且用通俗易懂的语言对它们加以诠释。在当前信息、通信系统飞速发展的大背景下，本书力图用较多的例子和图表来阐述概念和理论，同时尽量避免纠缠于烦琐难懂的公式证明之中。为了加深读者对所讲述知识的理解，每章最后都配有适量的练习题供读者选用。

本书可作为高等院校电子信息类学生双语教学的教材或参考书，也可作为通信、电信、电子等领域从业人员的参考资料。

This book is mainly about foundation of information theory and the coding theory. Also, some practical applications of these theories are mentioned in some occasions. On the basis of introducing the measure of information, it mainly focuses on the introduction of the theory of lossless source coding, limited loss source coding, channel coding and their applications. It pays attention to basic concepts and uses popular text to explain them. In the background of the current communication system, it includes many examples and figures to elaborate these concepts and theorems. At the same time, it does its best to avoid getting entangled in proving the equations. To deepen the understanding to the knowledge it touches upon, there are some exercises attached at the end of every chapter.

This book can be used as the teaching material for college students, and also, it can be the reference for career man in the area of communication, telecommunication and electronic.

Chapter 1 Introduction

Contents

Before it starts, there is something must be known

1.1 What is Information

1.2 What's Information Theory？

   1.2.1 Origin and Development of Information Theory

   1.2.2 The application and achievement of Information Theory methods

1.3 Formation and Development of Information Theory

Questions and Exercises

Biography of Claude Elwood Shannon

Chapter 2 Basic Concepts of Information Theory

Contents

Preparation knowledge

2.1 Self-information and conditional self-information

   2.1.1 Self-Information

    2.1.2 Conditional Self-Information

2.2 Mutual information and conditional mutual information

2.3 Source entropy

    2.3.1 Introduction of entropy

    2.3.2 Mathematics description of source entropy

    2.3.3 Conditional entropy

    2.3.4 Union entropy (Communal entropy)

    2.3.5 Basic nature and theorem of source entropy

 2.4 Average mutual information

    2.4.1 Definition

    2.4.2 Physics significance of average mutual information

    2.4.3 Properties of average mutual information

 2.5 Continuous source

    2.5.1 Entropy of the continuous source (also called differential entropy)

    2.5.2 Mutual information of the continuous random variable

 Questions and Exercises

 Additional reading materials

Chapter 3 Discrete Source Information

Contents

 3.1 Mathematical model and classification of the source

 3.2 The discrete source without memory

 3.3 Multi-marks discrete steady source

 3.4 Source entropy of discrete steady source and limit entropy

 3.5 The source redundancy and the information difference

 3.6 Markov information source

 Exercise

Chapter 4 Channel and Channel Capacity

 Contents

 4.1 The model and classification of the channel

    4.1.1 Channel Models

    4.1.2 Channel classifications

  4.2 Channel doubt degree and average mutual information

    4.2.1 Channel doubt degree

     4.2.2 Average mutual information

     4.2.3 Properties of mutual information function

     4.2.4 Relationship between entropy, channel doubt degree and mutual information

  4.3 The discrete channel without memory and its channel capacity

  4.4 Channel capacity

     4.4.1 Concept of channel capacity

     4.4.2 Discrete channel without memory and its channel capacity

     4.4.3 Continuous channel and its channel capacity

Chapter 5 kossless source coding

  Contents

  5.1 Lossless coder

  5.2 Lossless source coding

     5.2.1 Fixed length coding theorem

     5.2.2 Unfixed length source coding

  5.3 Lossless source coding theorems

     5.3.1 Classification of code and main coding method

     5.3.2 Kraft theorem

     5.3.3 Lossless unfixed source coding theorem (Shannon First theorem)

  5.4 Pragmatic examples of lossless source coding

     5.4.1 Huffman coding

     5.4.2 Shannon coding and Fano coding

  5.5 The Lempel-ziv algorithm

5.6 Run-Length Encoding and the PCX format

Questions and Exercises

Chapter 6 Limited distortion source coding

Contents

6.1 The start point of limit distortion theory

6.2 Distortion measurement

   6.2.1 Distortion function

   6.2.2 Average distortion

6.3 Information rate distortion function

6.4 Property of R(D)

   6.4.1 Minimum of D and R(D)

   6.4.2 Dmax and R(Dmax)

   6.4.3 The under convex function of R(D)

   6.4.4 The decreasing function of R(D)

   6.4.5 R(D) is a continuous function of D

6.5 Calculation of R(D)

   6.5.1 Calculation of R(D) of binary symmetric source

   6.5.2 Calculation of R(D) of Gauss source

6.6 Limited distortion source encoding theorem

Additional material for this chapter

Questions and exercises

Chapter 7 Channel Coding Theory

Contents

7.1 Channel coding theorem for noisy channel

7.2 Introduction: the generator and parity-check matrices

7.3 Syndrome decoding on q-ary symmetric channels

7.4 Hamming geometry and code performance

7.5 Hamming codes

7.6 Cyclic code

7.7 Syndrome decoding on general q-ary channels

Questions and exercises

Bibliography